Running a Green CI Pipeline Isn't the Hard Part Anymore

Six months ago, a team introduced an AI coding assistant to their software development process. Initially, pull requests doubled, and generated code entered branches faster than reviewers could evaluate it. However, the continuous integration (CI) pipeline struggled to keep up, with build times extending to 38 minutes and 4,000 tests running on every commit, even though only 60 were relevant. This shift from human-generated code to machine-generated code at machine speed presented new challenges for the CI pipeline.

The original goal of the CI pipeline was to assist developers writing code. However, with AI-generated code and the corresponding volume of commits, the bottleneck shifted to post-commit tasks such as dependency resolution, compilation, test execution, and image builds. These processes remained unchanged, running at the same pace as before the AI assistant's introduction. Consequently, developers faced significant waiting times for builds, similar to when they submitted only a couple of pull requests per day.

A survey conducted by GitHub in 2023 revealed that developers spent as much time waiting for builds as they did writing code. Incredibuild's survey found that 98% of developers wasted time waiting for builds, with build times increasing nearly 16% year over year, even before AI-generated commits became a factor. Additionally, the trust in the CI pipeline was compromised due to an increased number of commits and the associated higher exposure to flaky tests.

For instance, at Slack's mobile team, flaky tests drove their main branch build pass rate down to 20%. Each failure took 28 minutes to triage, causing critical delays in the merge queue.

To address these challenges, two approaches emerged in the AI era: faster execution or less work. The first approach involved deploying bigger runners, parallelized tests, cached dependencies, and conditional pipeline logic. While these optimizations helped, they primarily targeted the wrong layer of the pipeline. Doubling CPU resources on 5,000 tests still resulted in running 4,700 unnecessary tests, regardless of whether the code was written by humans or machines.

The more promising approach was to stop performing unnecessary work altogether. This involved detecting which dependencies had changed, which modules required recompilation, and which tests were affected by the code changes. By skipping redundant work, teams could achieve significant savings in build times. Harness CI, a continuous integration platform, offers three intelligence features to help mitigate these issues: Cache Intelligence, Test Intelligence, and Build Intelligence.

These features operate at the stage level, allowing teams to enable them within their pipeline configuration without the need for plugins or custom caching scripts.

Cache Intelligence automatically preserves and restores dependency caches between runs, eliminating the need for manual configuration or complex cache invalidation scripts. Test Intelligence, on the other hand, selectively runs unit tests based on code changes, using a call graph to identify affected tests and skipping the rest. Lastly, Build Intelligence focuses on testing only the necessary parts of the build process, such as rebuilding only the affected Docker layers or skipping full test suites for small code changes.

Implementing these intelligence features in a Java Maven pipeline using Harness Cloud demonstrates the potential for significant improvements. By enabling Cache Intelligence and Test Intelligence, developers can skip unnecessary re-downloads of dependencies and run only the relevant tests. In practice, this resulted in the pipeline saving 13 MB of Maven dependencies and restoring them in just 9 seconds on subsequent runs.

Similarly, Test Intelligence reduced the number of tests executed from 4,000 to just 60, focusing solely on the impacted tests and their related tests.

In summary, while AI coding assistants have transformed the development process by writing code faster, the CI pipeline has not evolved at the same pace. The bottleneck now lies in post-commit tasks, which remain unchanged and continue to consume significant time and resources. By embracing intelligent optimizations such as Cache Intelligence, Test Intelligence, and Build Intelligence within platforms like Harness CI, teams can significantly reduce waste and improve overall efficiency in their development workflows.