Faster CI with Orchestrated Pytest Workers
Close, a company that uses nearly 58,000 automated tests for its primary backend service, found that running all of those tests in CI took under 3 minutes. They had been using Jenkins and nose testing library since 2012 and later switched to CircleCI, GitHub Actions, and pytest. While the general shape of a test run worked well for small applications, as the application grew, the run tests step became slow, taking tens of minutes.
The team then turned to parallelism to speed up the tests. They found multiprocessing parallelism and deterministic work splitting as two common options. However, multiprocessing didn't work for them since many tests relied on external databases, leading to logical conflicts. Deterministic work splitting was a more attractive option, as it added only minor CI configuration and required installing a plugin and adding a few flags to the CI command.
This setup has been relatively easy to set up for a long time, with test splitting introduced in late 2014. Over the years, the team improved the CI speed through minor incremental changes, but eventually hit a wall. Even at 17x parallelization, CI times grew longer. The team realized that running test collection could take up to a minute, which was a fixed-cost step and scaled with the size of the entire app/test suite.
To address this, they looked at currents.dev, which maintains a centralized queue of work and tweaks the test runner logic to consume from that queue instead of a deterministic -split of work. This approach offers two major unlocks: smart orchestration, which splits workloads evenly, and spot instance support, which redistributes work if a worker's node is taken by AWS.
By implementing these concepts in pytest, the team was able to scale all fixed-cost setup steps effectively. They reduced 17 on-demand runners to 2 on-demand runners and 54 spot instances, achieving significant cost savings while maintaining CI speed.
Written by urgent.news from Lobsters's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.