The 2 AM Silent Failure: What Running AI Agents in Production Taught Me About Stability

In the world of AI agents, failures often occur in unexpected ways. Rather than experiencing a noticeable crash during a demo, these agents tend to falter later, in a more subtle manner. A task may run during the early hours of the morning, fail quietly without alerting anyone, and the user only discovers the issue the following day – a full day of work potentially lost.

At MeshCtx, we have encountered such failures in our own production system, running on a small three-machine cluster. To gauge our stability, we examined the health check from one of our production instances, which reported 15 out of 15 modules online with zero errors and was running on version 3.121.7. The source of this stability lies in several key factors.

Firstly, we have a robust set of automated tests to support our system. We are proud to share that we have 3,728 automated tests, all of which have passed successfully. These tests cover three different platforms: Windows, macOS, and Linux. Our best result on the LongMemEval EM test is 64.6% (3-sample best-of-3, compared to a 62.5% symmetric baseline).

Interestingly, the tighter the memory budget, the better the performance – for instance, at a 16KB memory budget, we achieved a 16.7 percentage point improvement over the symmetric baseline.

Secondly, stability means more than just not breaking. It means that our system has been tested thoroughly across the three platforms, reducing the likelihood of encountering the unforeseen bugs that might arise when deploying to production. Our extensive test coverage is not merely a cost line; it is a demonstration of respect for the user's time and resources.

Thirdly, our system is designed to remember. Most agent failures stem from forgetting, as one task's position list may bleed into another, or yesterday's task state might overwrite today's. Our solution is a layered memory approach with 17 regions, ensuring that positions lists do not bleed into article drafts, and yesterday's task state does not overwrite today's. Remembering the right things is the challenging part, but it is crucial for maintaining stability.

Finally, our system behaves consistently across all platforms, whether it is Windows at the office, macOS at home, or Linux in the cloud. We have automated this consistency rather than relying on manual restarts. For those looking to evaluate AI tools, we recommend checking whether the team publishes their test numbers. Teams that openly share their test results typically have something to back up their claims.

MeshCtx, which is free and open source under the MIT license, is available at meshctx.com. We invite everyone to run the tests and check the health endpoint, so you can verify our claims and those of others.