The wrapper that made my overnight AI scraper safe to leave running
I wanted a small research database to grow from one subreddit to ten communities where people talk about building with Claude Code and automation. Doing it properly was estimated at about $118. It cost about $34, and the corpus went from roughly 1,000 chunks to 5,846. Overnight, while I slept. The scraping ran through a paid provider (Apify) that charges per delivered record. The cleaning,…
A researcher aimed to create a database that expanded from one subreddit to ten communities discussing Claude Code and automation. The initial cost was estimated at around $118, but it only amounted to $34, growing from 1,000 chunks to 5,846. The scraping process ran overnight using a paid provider (Apify), while the cleaning, tagging, and indexing were done using Python. A PowerShell script, written by Claude, was the crucial element ensuring the entire process remained safe to run without constant supervision.
The script had three essential rules. First, every phase needed to be resumable, keeping its own state of chunks processed, spent money, and recovered content. This allowed the script to handle failures, such as Reddit blocking the scraper for hitting too many communities in a single session. The fix was to batch scrape three subreddits at a time, and if any failed, they would go into a retry queue. This way, the script could retry without user intervention.
Second, success should be measured by the provider's delivered-record count, not by the user's own count. The script compared the number of records delivered by the provider against a minimum threshold for each community, ensuring that even if the user didn't notice any change, the operation could still be considered successful.
Lastly, the script reported everything, including failures. If an error occurred during the process, a coverage table, status banner, and a report would still be generated, providing valuable information for troubleshooting. The final report helped distinguish between a successful run and a failed one, even if they appeared identical at first glance.
The researcher encountered two main issues: Claude Code's headless output and an incorrect table generation. To address this, they started judging the agent's performance based on timestamps and CPU usage rather than the log file. They also changed the method of creating tables to count the expected number of rows instead of visually inspecting the output. These improvements made the script more reliable and trustworthy. The script is freely available on GitHub, allowing others to adapt and use it for their own purposes.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
