{
  "id": 1993008,
  "title": "Case Study: A Free Model Wrote a C++ Tree Hasher. The Reference Oracle Found Three Bugs.",
  "url": "https://urgent.news/2026/08/19/case-study-a-free-model-wrote-a-c-tree-hasher-the-reference-oracle",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-19T18:35:15.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/datacpp_8185/case-study-a-free-model-wrote-a-c-tree-hasher-the-reference-oracle-found-three-bugs-dm1"
  },
  "original_language": "en",
  "account": "Conclusion: a free model created a functional C++17 directory hasher in one attempt. Although the draft compiled and ran, it still contained three real bugs. These bugs were identified before the tool ever interacted with a production cache using a differential test. Generation was the easy part, while verification was the crucial deliverable. The background involved needing a consistent directory tree hash for cache invalidation in a small build pipeline. Writing the tool manually would have taken approximately 200 lines of C++ code. The challenge lay in handling ordering, symlinks, and metadata that could affect the hash. To experiment, the author utilized MonkeyCode's free model access and server option, while conducting verification on their local laptop. The plan was to have the model generate the initial version, then validate it against a reference oracle. The contract stated that the same tree should always produce the same hash, regardless of the machine, while different content, name, or symlink target should result in a different hash. Metadata such as modification time (mtime) and inode should not impact the hash. Implementation began with crafting the prompt, where the model was provided with the contract, C++17 standard, and the requirement of using only the standard library. The model produced a single .cpp file that compiled on the first try. However, the process did not end there. A reference oracle was built to compare the tool against a standardized shell pipeline that normalized the directory tree into a sorted stream of records and hashed the stream. This pipeline was slow but unambiguous, serving as the reference oracle. A script was then used to generate 1,000 random trees with various scenarios, including duplicate names in different directories, symlinks, and files with identical content. Upon running the differential test, three bugs were found. Firstly, the tool produced different hashes on different machines due to unspecified ordering in the recursive_directory_iterator. The fix involved collecting paths, sorting them, and then hashing. Secondly, retargeted symlinks produced identical hashes, leading to the same hash being generated for different inputs. The solution was to hash the link target string instead. Lastly, a change in the modification time of a file resulted in a new hash, which was corrected by only considering the path, file type, and content when generating the hash. After implementing these fixes, the tool matched the reference oracle for all 1,000 trees. A second batch of 500 trees with deeper nesting and longer paths was also tested, with zero mismatches. The model's draft was completed in under a minute, while the three fixes took about an hour, including the creation of the harness. The harness not only caught the three bugs but also identified two issues in the author's own fixture generator. The oracle's role was to evaluate the model's output rather than the author's. Interestingly, the model did not comprehend the contractual requirements related to determinism across machines, which were the root causes of the three bugs. Ultimately, the model generated the code, but it was the 30-line harness that identified the errors and pinpointed the exact failing inputs. The lesson learned was that compiling and running the code is not enough; the model must adhere to the actual contract. The use of a reference oracle transformed the review process into measurement, highlighting the importance of verification over mere generation. The experiment demonstrated that separating code generation from verification creates a hard gate that must not be bypassed. The model ran remotely, while the oracle ran locally, emphasizing the importance of aligning the two. The model did not fully grasp the implications of determinism across machines, which was the core problem. Therefore, it was essential to ensure the model understood the contract thoroughly. The limitations of this workflow include the reference pipeline's inability to account for ACLs, extended attributes, and hard links, which were not considered in the tool. If these factors are relevant to the specific use case, the oracle should be designed first, followed by the tool's development. The free model also had a blind spot, assuming local consistency without explicitly questioning what \"deterministic\" meant across machines. This assumption was the underlying issue. In summary, the generated code was not the key artifact; instead, the 30-line harness that identified the errors and directed attention to the input was the crucial element. If attempting this workflow with a free model endpoint, it is recommended to build the oracle before examining the model's output. The free tier is sufficient for running the experiment; the gate, which ensures verification, is what makes the process valuable.",
  "summary": "Conclusion first: a free model drafted a working C++17 directory hasher in one pass. The draft compiled, ran, and was still wrong. A differential test against standard system tools found three real bugs before the tool ever touched a production cache. Generation was the cheap part. Verification was the deliverable. Background I needed a deterministic hash of a directory tree. The use case was…",
  "key_points": [
    "Free model generated C++17 directory hasher in one attempt",
    "Three bugs found via differential test before production use",
    "30-line harness identified errors and pinpointed failing inputs"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}