I replayed 7 repos' READMEs in fresh clones. 2 of them fail their own quickstart.
Your README is a promise nobody tests. Setup instructions rot silently. You rename a script, add an env var, switch a package manager, and the quickstart keeps promising a path that stopped working weeks ago. Nobody on the team notices, because everyone who would notice already has the repo running. The only people who hit the wall are new users, and they bounce without filing an issue. I got…
The author of this tool discovered that 2 out of 7 README files failed their own quickstart after cloning and testing them. One particularly problematic README instructed users to run "npm run start," but no corresponding "start" script existed in the package.json file. This error led new users to the README page, become stuck at step 2, and ultimately leave, resulting in a lost star rating for the repository.
The tool, called "coldstart-cli," works by recreating the steps outlined in the README without executing any actual commands. It first checks the repository state in a temporary directory, ensuring no node_modules or .env files that could skew the results. The tool then parses the shell commands from the README, replaying them one by one while tracking the working directory and environment changes.
The author found that this approach revealed several issues. Some READMEs listed scripts that were not present in the package.json file. Others contained commands that referenced files not included in the repository, which users would be unable to provide. Additionally, the tool flagged commands that executed on the reader's machine rather than in a sandboxed environment.
One particularly egregious example was a README that promoted an "npm run start" command, but the package had no "start" script defined. Users attempting this step would be greeted with an exit code and the last lines of output, indicating a failure before the app even had a chance to initialize.
The tool's limitations are intentional. It focuses solely on whether the application can boot, not if it functions correctly. It also skips over placeholder arguments like API keys, providing a note instead of failing the entire process. The tool runs in a clean container with Node 18+, requiring no additional dependencies to operate.
The author encourages others to test their own READMEs using this tool, as they are particularly interested in discovering unexpected issues that could break the user experience. By providing a clean, reproducible environment, coldstart-cli aims to expose hidden bugs in README documentation and promote better practices for onboarding new users.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
