When Your AI Reviewer Remembers Too Much: A Two-Phase Memory Probe
Most AI code-reviewer evaluations treat the candidate as an amnesiac: feed it one pull request, read one verdict, and move on. Persistent-memory reviewers break that model because they keep history across PRs, and that history becomes a second source of bugs. The dominant failure is no longer amnesia but overconfidence in stale context. A two-phase probe exposes whether a candidate trusts its own…
AI code-reviewer evaluations typically see candidates as if they had no memory at all. Each pull request is evaluated individually, with no regard for previous interactions. This lacks the ability to consider the broader context, leading to potential oversights. A new approach called a two-phase memory probe seeks to uncover whether an AI reviewer relies too heavily on outdated information.
A repository has been prepared to test this phenomenon. This fixture illustrates a conflict between two modules: the legacy metrics_service.py and the newer telemetry_service.py. The documentation includes an ADR-0012 decision that officially renamed the metrics pipeline to telemetry. The probe contains two phases.
In Phase 1, the AI is given a boring task of adding retry logic to the telemetry_service.py file, alongside updating a configuration file. This phase allows the AI to observe the repository's history and understand the existing conventions. The results from this phase are not graded.
Phase 2 presents a different scenario. The AI must delete the legacy metrics_service.py file, move the telemetry_service.py module to the canonical module, and introduce a deliberate bug. The AI must now use the decision from ADR-0012 to correctly handle the renamed module and recognize the seeded bug. A proper review must block on the missing guard in the transmit function while accepting the rename and citing the decision.
The candidate's review will be judged based on three categories: blocking, consistency, and uncertain. The blocking category identifies any correctness issues, with references to the specific lines of code. The consistency category checks if the AI's decisions align with the latest decisions in the repository. The uncertain category identifies any claims that cannot be verified.
Scoring the review is simple with an HTTP mapping system. A 200 status code (90-100) signifies that the AI correctly found the seeded bug, accepted the rename, and provided citations from the decisions. A 301 status code (60-89) means the AI recognized the change but flagged it as unnecessary churn. A 409 status code (30-59) indicates that the AI insists on retaining the old namespace, contradicting the decision. A 404 status code (0-29) means the AI missed the seeded bug and did not make any file-level claims.
A shell script provides a zero-cost runner, allowing the memory probe to be executed easily. The script clones the fixture repository, sets up phases for the probe, and runs the reviewer CLI to evaluate the review. The scorer is a simple Python script that parses the JSON output from the reviewer and calculates the score based on the reviewer's response.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
