Extracting Static Public Data with Python (Zero Dependencies)
If you need to extract basic structured data from a static, publicly accessible website, it can be tempting to immediately reach for frameworks like Selenium, Scrapy, or Playwright. However, for simple static pages, Python's standard library can often handle the job without installing any external dependencies. In this tutorial, we'll build a lightweight data extraction script using only urllib ,…
Static websites often contain structured data that can be extracted using Python's built-in libraries, without the need for additional dependencies. This tutorial demonstrates how to create a lightweight script using the urllib, re, html, csv, and logging modules to extract specific data from a public website. The approach covers fetching HTML content, parsing it to extract desired patterns, cleaning HTML entities, and exporting the results to a CSV file.
It also includes basic logging and error handling to ensure smooth operation. The tutorial cautions that this method is limited to extracting data from static, publicly accessible pages and does not support rendering JavaScript, handling infinite scrolling, bypassing CAPTCHAs or anti-bot protections, accessing login-protected pages, or crawling multiple pages automatically.
It is crucial to ensure you have permission to perform automated extraction on any website. The code example focuses on extracting quotes and author names from the public scraping sandbox quotes.toscrape.com, with the process divided into separate functions for fetching the page, parsing the HTML, extracting the data, and exporting it to CSV.
The script begins by setting up logging and defining a PublicDataScraper class, which takes a target URL as input and initializes an empty list to store extracted data. The fetch_page_html method retrieves the HTML content from the target URL, handling potential errors and logging the process. The parse_html method uses regular expressions to find and extract quotes and author names from the HTML, cleaning the data and organizing it for export.
Finally, the extract method runs the overall extraction process, logging each step and exporting the results to a CSV file if data has been successfully extracted.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.