Agent stdout Is Not Your Test Plan

Many people mistakenly believe that checking for "All tests passed" in an agent log proves the test suite ran successfully. However, this is not a reliable method, as logs can be deceptive. Logs can fabricate results, reuse old snippets, or present an intended outcome as though it were actual output. To avoid this, treat agent logs as unverified speech, not a complete test run.

Another common misunderstanding is that an exit code of 0 indicates a completed change. This is incorrect, as various scenarios can generate a 0 exit code, such as a formatter returning 0, a skipped pytest session, or a --dry-run flag. To accurately assess the situation, you need to capture four specific fields: the actual command invoked, the working directory, the numeric exit code, and the hashes of stdout and stderr. Without these four data points, you cannot confidently determine the test suite's outcome.

Additionally, relying on logs that appear complete can be misleading. Logs exist in a buffer, while the repository exists on disk, and they can quickly become out of sync. An agent may generate a diff that it never wrote or create files it never mentioned. Instead, consider git status and git diff as reliable votes on the project's state. If the current HEAD is unknown, you are not reviewing an actual change, but rather an impression or "vibe."

Finally, relying on a single successful run on a scratch machine does not guarantee portability to CI environments. The scratch host is a controlled lab, and CI runs in a different environment with potentially different PATH, lockfiles, and OS packages. Instead of using agent logs as proof, focus on recording and storing reproducible artifacts.

Develop a script that treats agent speech as untrusted and serves as the final gate after an agent session. This script should ignore English and focus on recording verifiable votes, such as git status and git diff, rather than relying on potentially misleading log output.