Unique ID: 2015009

Division: Analysis/ Quality Management/ Methods
Issue Date: February 13th 2019
Last modified: February 22nd 2019
Collaborative

Automatic price collection on the internet: Using web scraping and web Crawlers for price index compilations

Using web Crawlers for price index compilations

Development and implementation of automatic price collection procedures using Web Crawlers for price index compilations. Assessment of automatically-collected data in terms of quality and efficiency gains and their applicability for other statistics.

Project Objective:

Pilot intended to go to production to supplement existing data, Pilot intended to go to production to replace existing data

Project Outcomes:

The automatic price collection project should support the objective to achieve higher quality (price) statistics while using existing resources more efficiently.

Statistical Area

Labor, Price

Project Sources
Project Sources
Type Of Institution: National statistical office
Big Data Source: Web scraping data
Region: Europe & Central Asia
Country Area: Austria
Id Country Regional: country
Partnerships
Partnerships
Partnership Comments: None
Data Coverage
Data Coverage
Data Coverage: Other
Cost Implication: Free
Project Details
Project Details
Frequency Comments: The coverage and frequency can be handled flexibly. In theory, all data of a website can be scraped daily. In practice, there is mostly no need for this so that coverage and frequency are reduced. However, daily web scraping of websites is being tested in the pilot project.
Data Quality
Data Quality
Quality Aspects Evaluated: Privacy and Security, Completeness, Usability, Time Factors, Accessibility, Relevance, Validity, Accuracy, including selectivity, Coherence, including linkability to other sources
Data Quality Concerns Comments: Internet prices might not be transaction prices. Unknown quantities of the sold products and services offered on the Internet. Not easy to avoid double counting when scraping job vacancies.
Methodology
Methodology
Methods Used: Traditional statistical methods
Other
Other
Income Level: High-income
Iso: AT
Timeframe To Produce Indicator: NA
Frequency Comments: The coverage and frequency can be handled flexibly. In theory, all data of a website can be scraped daily. In practice, there is mostly no need for this so that coverage and frequency are reduced. However, daily web scraping of websites is being tested in the pilot project.
Write Your Own Review
You're reviewing:Automatic price collection on the internet: Using web scraping and web Crawlers for price index compilations
Your Rating