A Green Test Suite Can Still Hide a Broken PR: 5 Questions to Ask Before Merging

AI coding tools can swiftly generate code and automated tests. However, both may seem correct while crucial functionality remains untested. A passing suite of tests offers evidence, but it doesn't guarantee correctness. When considering a pull request (PR) that AI helped create, developers should ask five key questions to ensure the tests offer sufficient assurance before merging.

Firstly, developers should examine what specific behavior the test is verifying. It's not enough to check the number of assertions. For instance, consider a scenario where a payment service should reject transactions when an account lacks sufficient funds. A test might only confirm the service returns a response, without verifying that the payment was indeed rejected.

Developers should ask themselves what critical behavior the test is meant to protect - what outcome would confirm the behavior is functioning as intended? If they can't answer these questions, the test may offer less confidence than its green status suggests.

Secondly, they should assess whether the test can pass without examining anything meaningful. Consider a JUnit test that checks if a payment was processed, but only verifies that execution reaches the assertion, not that the payment was rejected. A more meaningful test would verify the actual outcome, using an assertion that aligns with the application's behavior. The critical principle is that the assertion should confirm the behavior the test claims to protect.

Thirdly, developers should consider if mocks are obscuring the behavior that truly needs verification. Mocks are useful when testing dependencies, failure conditions, and isolated units of code. However, excessive mocking can make a test appear relevant while avoiding the actual behavior that matters. For instance, a test that verifies an order is saved correctly might only check that the save() method was called, using mocks for the repository.

While this shows a method was invoked, it doesn't necessarily confirm the order was stored correctly or that the resulting state is accurate. Before approving heavily mocked tests, developers should ask: Is the test checking an outcome or just an interaction? Is the mocked dependency the behavior we're testing? Would an incorrect implementation still satisfy the assertions? Would an integration test better verify the interaction between components?

Fourthly, developers should verify whether the test truly covers the changes described in the PR. A PR might alter behavior without modifying the test that initially covered it. For example, a developer could change the conditions under which a user can access a resource. The existing test might still confirm successful access but fail to check the newly introduced restriction.

Developers should compare the change with the tests: Which behavior has changed? Which existing tests cover that behavior? What new failure scenarios could the change introduce? Are the expected outcomes clear? This step also requires considering the broader context of the repository, as sometimes the affected behavior might involve callers or related files not included in the PR diff.

Lastly, developers should ask if the test would detect a deliberately broken implementation. It's a useful exercise to imagine the implementation contains a defect that violates the requirement. Would the test fail in this case? If the answer is no, the test may not provide adequate protection for that requirement. Mutation testing takes this concept further by making deliberate changes to the implementation and checking whether the tests detect them.

While it requires additional effort compared to a standard test run, it can offer stronger evidence of test effectiveness.

In conclusion, before merging a PR - especially one generated with AI assistance - developers should carefully evaluate their tests. Do the tests verify the intended behavior? Could they pass without a meaningful assertion? Are mocks obscuring the true behavior to be verified? Do the tests cover the changed behavior and important regression scenarios?

Would the tests catch a broken implementation? These checks won't guarantee that every defect is caught, but they can help identify tests that provide confidence without sufficient evidence. While AI can accelerate code and test creation, the review process still needs to determine what those tests actually prove.