How I caught an AI-generated Python regression that the tests missed -hackathon project
The change was small, the diff looked sensible, and the existing tests passed. Then I tried an empty list. The original function returned []. The revised version returned None. Nothing immediately crashed, but code expecting a list had quietly received a different contract. That gap is why I built Cross‑Examine. Cross‑Examine is a verification harness for Python changes. Given a base Git revision…
The Python regression slipped past the tests unnoticed. I caught it by comparing the expected and actual outputs when an empty list was passed. The original function returned an empty list, but the modified version returned None. This discrepancy caused code that expected a list to behave unexpectedly. To prevent such issues, I developed Cross-Examine, a verification harness for Python code.
It captures the behavior from the original code and compares it to the modified version. If the behaviors differ, Cross-Examine records the input, command, and outputs from both revisions. The tool then presents a verdict indicating whether the change is broken. Cross-Examine relies on GPT-5.6 to propose checks, while the deterministic code handles the final decision.
It is currently limited to Python and specific call shapes. The tool runs the original code and verifies the modified code, making it not a sandbox. Cross-Examine should be used on trusted repositories only, as it executes target code. It was built during OpenAI Build Week using Codex and GPT-5.6, with Codex assisting in implementation and GPT-5.6 proposing checks.
The repository includes an offline demo that demonstrates the tool's capabilities without requiring an API key.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
