SWE-Gate: Why Passing Tests Isn't Enough for Agent-Generated Code

Coding agents excel at passing tests, but when it comes to code reviews, they fall short. The benchmark SWE-Gate addresses this gap by measuring both test success and review compliance. Derived from actual pull request comments, SWE-Gate introduces a second gate in evaluation: does the patch satisfy the review rules that would prevent merge in production?

Out of 644 agent-generated patches that cleared functional tests, 221 faltered on review constraints, amounting to a 34% false-positive rate when solely assessing test passage. While repository-level benchmarks like SWE-bench gauge an agent's ability to generate a patch that passes existing tests, they overlook the crucial second gate: human review.

Review constraints encompass style and formatting rules, architectural patterns, security boundaries, performance expectations, and maintainability requirements. These guidelines reside in review processes, team norms, and the tacit wisdom of senior engineers—information not encoded in test suites. SWE-Gate constructs 303 repair instances across 75 Python repositories, each comprising a functional test suite, a review constraint test suite, a non-compliant patch passing tests but failing review constraints, and a gold patch meeting both criteria.

The benchmark explicitly separates functional correctness evaluation from constraint compliance evaluation, presenting the agents' score as a tuple of these two outcomes. The authors mined review comments from merged pull requests, filtering for change requests and extracting enforceable rules as constraints. Examples include adding type hints, using existing helpers instead of regex, and extracting logic into separate methods.

Each rule becomes a constraint test, executed after functional tests pass. The architecture of a two-stage evaluation harness includes two distinct test runners: one for functional tests and another for review constraint tests, ensuring constraint tests run only after functional tests succeed. The key takeaway: constraint tests must follow functional tests to distinguish between an agent's inability to solve the problem and its suboptimal solution.

Four LLM backends powered a common coding-agent scaffold in the SWE-Gate experiments. Failure modes revealed that 35.8% of failures stemmed from functional tests, while 22.0% were due to review constraints—most often missing type hints, duplicated code, incorrect abstraction layers, hardcoded values, and incomplete docstrings. These are everyday review friction points not captured in test suites.

For agents intended for production, the two-stage gate necessitates a nested-loop feedback loop: Issue → Agent → Patch → Functional Tests → Pass? (No: retry) → Yes: Constraint Tests → Pass? (No: retry) → Yes: Completion. This approach ensures agents produce merge-ready code by addressing both functional correctness and review constraints.