We put 7 frontier models on call for 100 real Kubernetes incidents. The best fixed 79.
TL;DR: Tymfi Labs is a benchmark for AI SRE agents. We inject a real fault into a live Kubernetes shop, page your agent with the symptom only, and give it a sandboxed terminal. A pass means the live system works again afterwards. What the agent writes in its report doesn't count. On our current edition (100 incidents, 8 layers), the best frontier model fixed 79%. You can connect your own agent…
Tymfi Labs has evaluated seven frontier models on 100 simulated Kubernetes incidents, with the best model fixing 79% of the issues. The testing process involves injecting real faults into a live Kubernetes environment, giving the AI agent a sandboxed terminal to diagnose and resolve the problem. Factors like tool usage, command budget, and time limit are kept consistent across models.
The top performers, Gemini and Argon, showed similar pass rates, with a cost difference of about 3x between the best and worst models. However, even the best model failed to fix incidents in about 20% of cases. The evaluation highlights the limitations of transcript-based grading systems, as they cannot detect when an agent partially fixes an issue and then fails to address subsequent problems.
Tymfi Labs offers the benchmark as an MCP server, allowing any MCP client to run the tests. Future editions will include more challenging multi-fault incidents and reinforcement learning training on the same scenarios.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
