When every check blocks, verifying a small change costs an hour

Your team's pipeline began as a swift process. The introduction of end-to-end tests, prompted by a genuine bug, and subsequent security reviews introduced additional checks. These checks were essential, and thus, nobody proposed their removal. Consequently, verifying a change involving two files now takes longer than the initial coding.

As more checks were added over a few months, verifying even small tasks took nearly an hour. The issue wasn't removing these checks or making them faster. Instead, the issue lay in redefining what each check blocked. Some checks had to block individual changes, while others only needed to block deployments. The combination of both led to the pipeline functioning like a bottleneck.

The quality gate was the crucial component that everything else depended on. The most time-consuming check was the model-assisted interface review, which involved an agent opening a browser, navigating the application, and assessing whether the implemented features looked and behaved as intended. Although valuable, it was incredibly time-consuming due to the need to start the environment, navigate screens, and interpret results.

Running this check on every task was akin to a designer reviewing the entire application every time someone made a code change. The key question was not whether the check was worth it but whether it was blocking a task or the deployment. The costs of these checks varied significantly, with the former occurring dozens of times a week and the latter only once per release.

Mixing these checks led teams to disable both over time. To address the problem, the team decided to create two tiers of checks. Tier 1 consisted of fast, deterministic, and task-specific checks, including the full backend suite, task-specific end-to-end tests, and a security trace. Tier 2 comprised slower, expensive, and batch checks, such as the full end-to-end suite and model-driven interface review.

The tier structure made it clear which checks blocked tasks and which blocked deployments. The implementation process involved decoupling the requirement of the interface review from the per-task gate, adding new Tier 1 checks, and finally removing the per-task signal. This sequence ensured that the system remained functional during the transition.

The gate could no longer demand the interface review if the project configuration didn't declare its existence, preventing unnecessary requirements from causing issues. The final outcome was successfully reducing the per-task gate from approximately an hour to minutes across all projects. The team also discovered that restarting the server led to the loss of the work plan, which became a new issue to tackle.