My CI perf budget failed twice on changes that couldn't be slower
The perf job failed on a change that couldn't have made anything slower. I reran it and it passed. It happened again on the next one. That is the moment a performance budget stops being useful: nobody trusts a red build that goes green when you click "re-run". I could have loosened the budget. Instead I kept it and made each number say whether the app or the machine was slow. This is how the…
The performance budget for a software application failed on changes that couldn't have made it slower. Despite the budget being in place, a red build turned green when runners were clicked. The author kept the budget and measured various aspects of the app, including opening a large workspace and propagating edits between two browsers. The app recorded user timing marks when the issue list was shown and propagated edits in real-time.
The budgets were set based on median times from five runs, with a median value being over the budget triggering a failure. Machine probes were added to identify when runner noise caused failures. The probe tracked CPU stalls and disk write delays and noted them in the results.
The edit test initially timed the handshake, but was modified to wait for the sync badge to say 'Live' before starting the clock. This change, along with the machine probe, helped identify when runner issues were causing failures. The budgets were set to ensure instant feel for reopening a large workspace and seeing someone else's edit, with a looser budget for propagation since runner noise dominated it.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.