Web Scraping

Web scraping means using software to collect information automatically from websites rather than copying it manually. A scraping program retrieves web pages, identifies the required content and extracts it into a structured dataset, such as a table of product names and prices.

How automated collection works

A scraper converts information presented on web pages into data that can be stored, searched or analysed.

• The program sends requests to a website and receives page content, commonly in HTML, the markup language used to structure web pages.
• It parses the content, meaning that it identifies elements such as headings, links, prices or table entries and extracts the selected values.
• The extracted information can be cleaned and saved in a spreadsheet or database, and the process can be repeated across multiple pages.
• Some scrapers use browser automation to retrieve content that appears only after scripts run or a user interacts with a page.

Scraping, crawling and APIs

These are related methods of accessing online information, but they perform different functions.

• Web crawling primarily discovers and visits pages by following links; scraping extracts particular information from those pages. A single system may do both.
• An application programming interface (API) provides a defined way for software to request data, often in a structured format. Scraping instead extracts information from web content and may fail when the page layout changes.
• Scraping can support tasks such as price comparison and research; whether a particular use is permissible depends on the data, access conditions and applicable law.

Intellectual property and responsible use

Public accessibility is not the same as permission to reuse content. Collecting information and reproducing or commercially redistributing protected material raise distinct questions.

• Under India's Copyright Act, 1957, literary works include computer databases, and copyright protection applies to original literary works. Whether particular material is protected and whether its use infringes copyright require separate assessment.
• Copyright does not protect mere facts or ideas, but original expression and an original selection or arrangement of material may be protected.
• Responsible collection should consider website access conditions, copyright permissions, personal-data concerns and the burden that repeated requests place on servers.