Why API Test Generation Is a Judgment Problem, Not a Code Generation Problem

When KushoAI began utilizing large language models for API test generation, the outcomes appeared promising at first. Tests were produced rapidly, and the breadth of coverage expanded. The team was enthusiastic. However, upon closer examination, it became evident that the model generated an excessive number of tests, missed crucial boundary cases, and failed to check for meaningful information within responses.

Although the tests were valid, the resulting test suite lacked usefulness. This article delves into the lessons learned during the pursuit of a more effective approach. It explores how the team diagnosed the underlying issues, why conventional fixes such as improved prompting prove insufficient, and the architecture that emerges when test generation is perceived as a judgment problem rather than a code generation problem.

The six types of failures that arise from fast test generation were identified through extensive analysis of thousands of real APIs. Over-generation, under-generation, redundancy, inconsistency, weak assertions, and cross-field relationship failures were all observed as distinct forms of judgment errors rather than knowledge gaps.

The initial reaction to these issues was to enhance prompting techniques. Structured prompts, multistep chains, context injection, explicit output schemas, few-shot examples, and detailed instructions about redundancy, coverage breadth, and assertion specificity were employed. While prompting did improve the structure and reduced obvious redundancy, its effectiveness plateaued.

It primarily enhanced exhaustiveness at the field level, leading to more missing-field tests, wrong-type tests, and boundary tests. However, it struggled to produce reasoning about the relationships between fields, business states, and workflow behavior. This critical aspect proved to be the most challenging failure mode. Cross-field relationship testing requires the model to understand how multiple valid fields combine to create an invalid state.

Prompting had limited success in addressing this complex issue. To resolve these challenges, the team adopted an architectural shift that treated test generation as two separate problems requiring distinct models. The first problem, judgment, involves determining which scenarios should be tested, identifying crucial edge cases, deciding what assertions are necessary, assessing the appropriate level of testing, and recognizing redundant tests.

This is a calibration problem that necessitates exposure to numerous examples of reviewed QA decisions across various API types. The second problem, mechanics, focuses on constructing valid payloads, generating executable tests, writing framework-specific output, and handling authentication context. General-purpose frontier models excel at handling this aspect.

By fine-tuning narrow models specifically for the judgment layer and utilizing frontier models for the mechanics layer, the team was able to effectively address the root causes of the problems encountered.