How to download all images from a website with Python (lazy-loaded, srcset and CSS backgrounds)
Right-click, "Save image as" does not scale past a handful of pictures. Here is a short Python script that downloads every image from a page, plus the traps that make naive scripts miss images or get you blocked. The script Install: pip install requests beautifulsoup4 . Run: python grab_images.py https://books.toscrape.com/ images #!/usr/bin/env python3 """ Download every image from a web page.…
The provided Python script is designed to download all images from a given web page, including lazy-loaded images and images specified in CSS backgrounds. It requires the installation of the `requests` and `beautifulsoup4` libraries.
First, the script imports necessary modules such as `hashlib`, `os`, `re`, `sys`, `time`, `urljoin`, `urlparse`, `robotparser`, `requests`, and `BeautifulSoup`. It sets up a user-agent string `UA` for the requests and defines a regular expression `CSS_URL` to match CSS background URLs.
The `best_from_srcset` function is defined to find the largest image URL from a `srcset` attribute. It splits the `srcset` string by commas and iterates through each part, determining the width (`w`) of each candidate image based on the presence of a numeric value following the width (e.g., `480w`). The function then selects the candidate with the highest width.
The `find_images` function takes the HTML content of the page and the URL of the page as input. It creates a BeautifulSoup object to parse the HTML and initializes an empty list `found` to store the image URLs. The function then searches for `<img>` tags in the HTML and extracts the image URLs from various attributes such as `src`, `data-src`, `data-lazy-src`, `data-original`, and `data-lazy`.
It also searches for images specified in the `srcset` attribute, lazy-load attributes (`lazy`, `lazy-src`), and CSS background URLs.
The script additionally handles images specified in `<picture>` tags with `srcset` attributes and `<meta>` tags with `property=og:image` or `name=twitter:image`. It extracts the `content` attribute from these tags and adds them to the `found` list.
Next, the script processes CSS background URLs by finding all `<style>` tags in the HTML. It adds the matched background URLs to the `found` list using a regular expression `CSS_URL`.
After collecting all potential image URLs, the script creates an empty list `urls` to store the final list of image URLs. It iterates through each URL in the `found` list and checks if it does not start with `data:` (which indicates an inline data URI). If the URL passes this check, it joins the URL with the base page URL using `urljoin` and appends the result to the `urls` list.
Finally, the script returns a list of unique image URLs by converting the `urls` list to a dictionary and back to a list using `dict.fromkeys`. This ensures that duplicate URLs are removed from the final output.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
