I built a local-first linter for vague release notes

Release notes frequently contain vague language like "enhanced performance" or "fixed multiple bugs." While these phrases may sound conclusive, they fail to inform readers about the specific changes, impacted users, or if any action is necessary. To address this issue, I developed ReleaseProof, a tool designed to evaluate the specificity and accuracy of release notes.

It takes two inputs: change evidence, such as pull request titles, issue summaries, test results, or concise diff summaries, and the draft release note meant for end-users. The checker then examines the provided text for vague assertions, missing audience details, missing migration instructions, insufficient concrete evidence, and claims without proper support.

For instance, if the evidence shows that a dashboard query count decreased from 18 to 6, a draft stating "Improved dashboard performance" would be flagged for lacking specificity. ReleaseProof operates as a linter, not a writer, meaning it does not create a revised changelog or make up information. Instead, it provides targeted prompts about missing facts, with the ultimate responsibility resting on the author to ensure the information's accuracy.

The current version of ReleaseProof functions entirely within the browser, requiring no account and not transmitting any data to external servers. It does not assess code correctness, security, or production readiness; its sole purpose is to scrutinize the clarity and relevance of the written content based on the supplied evidence.

After applying deterministic rules to 20 examples covering various scenarios like minor fixes, performance claims, breaking changes, migrations, fields, routes, and missing evidence, I further tested the tool using five actual release notes from projects such as uv, Ruff, pnpm, and Bun. This additional testing revealed several issues in the initial rule set, including the inability to recognize technical identifiers as concrete details, developers being forced to include an audience in their notes, reliance on literal word matching for evidence verification, and the potential for link-only releases to achieve high scores.

These identified problems have become regression tests. Upon evaluating four content-rich release notes, the tool successfully determined that they were specific, while a link-only example was appropriately flagged for not clearly stating a tangible change. My objective is to determine the practical utility of ReleaseProof. I am searching for a limited group of individuals who write release notes, changelogs, or user-facing PR summaries to test the tool using a sanitized example.

If you utilize this service, I would particularly appreciate your feedback on which suggestions were helpful, any false positives or missed issues, whether you modified the draft based on the feedback, and if you would consider using the checker again in the future. You can attempt ReleaseProof without creating an account and provide structured feedback through the link adjacent to the results.