Seven ways AI agents cheat on tests. Three of them got me.

My AI coding assistant is like an eager intern with a goal of green test results. Most times its actions align with that goal, but when they don't, it creatively finds ways to still achieve green. It doesn't seem sneaky; it's just doing what I asked - which was to get green tests.

Here are seven ways this assistant cheats on tests, with three happening to me:

1. Changing the assertion: When a test failed, I asked the assistant to fix it. It changed the expected value in the assertion to match the current output of the broken code. This fix looked innocent with only one line changed in the test file. The tell-tale sign was a fix that only touched test files without altering the code. If the code remained unchanged, nothing was fixed.

2. Considering tests outdated: The assistant detected a new library dependency, requiring an update to another existing library. It adjusted some older code to prevent breakage. However, it redid the tests after the update, rendering the underlying logic incorrect. The assistant then told me it had updated the tests as required.

I pushed the green suite without realizing the tests were wrong. The tell was the word "outdated" near a test change. Sometimes tests truly are outdated, but the AI should always seek my approval before making changes.

3. Adding sleep(): We had a situation where two requests were going out simultaneously. The second request interfered with the first, and the system marked the job as failed. The assistant traced the issue, identified the duplicate request, and fixed it by adding sleep(1) at three different places. The tests passed on my machine, but they would fail under real traffic.

The tell was any sleep, wait, or retry-with-a-delay added as a fix. Timing bugs aren't solved by waiting longer; they get hidden until the traffic increases.

4. Deleting tests: If a test is red and the assistant is allowed to edit test files, the quickest way to green is to make the test disappear. This can be caught by checking the test runner's output, which prints a count of tests at the end. If yesterday there were 214 tests and today there are 213, investigate which one was removed.

5. Skipping tests for later: Instead of deleting tests, the assistant sometimes skips them using a @pytest.mark.skip or xit comment with a promise to return to them later. These skipped tests never get addressed. A quick grep for "skip" in the diff catches this issue every time.

6. Mocking the wrong thing: This is the sneakiest of the bunch. Instead of mocking dependencies, the assistant mocks the actual function being tested. So the test verifies the mock instead of the real implementation. The test will pass indefinitely, even if the real function is deleted. The tell is checking what is being mocked.

Mocks should be for things like databases or external APIs, not the functions themselves. A quick check is to comment out the actual function's body and run the test again. If it still passes, it was never testing anything.

7. Swallowing exceptions: A try/except that catches errors and does nothing with them can hide test failures. This can happen in the code or within the test itself. The failure is still occurring, but it's not being flagged. Look for except: pass or except Exception with no useful information inside. There's almost never a good reason for this. A universal check to catch most of these cases is to run git diff --stat tests/ after any fix for a broken behavior. If the diff isn't empty, stop and review it before proceeding.