The 45-Minute Exit Drill: What Breaks When Your Free AI Server Vanishes

At 2:47 AM, an email arrived informing that the free allowance would expire in 72 hours. The demo continued to function, but after three days, all these would become inoperable. The author, having experienced both sides of this issue, decided to conduct an exit drill to test the resilience of their application when the free server vanished.

The drill was designed to be completed in 45 minutes using a single laptop and without any meetings. The goal was to make the application work without the free server within an hour, using only the tools available on the machine. The author chose a Friday afternoon to start the drill and set a timer to keep track of time.

The first five minutes were spent inventorying the dependencies. The author used a grep command to find all instances of client libraries used in the codebase. In this case, the damage was contained to one config file, two modules, and a test fixture that had hardcoded the remote URL. The fix was a single environment variable, but the five minutes spent searching revealed that if the endpoint URL was present in multiple files, the drill had already failed. It should be defined as an environment variable to prevent this issue.

The next 10 minutes were spent setting up a local replacement for the remote endpoint. In this case, MonkeyCode's open-source nature made this process relatively straightforward. The author cloned the MonkeyCode repository, followed the self-host instructions, and started the server using Docker. However, they noted that this process can be frustrating if a GPU is required or if a CPU-only local model is used, which would be slower and less accurate than the managed server.

After 15 minutes, the author redirected the traffic to the local server by setting an environment variable to the local endpoint URL. If the code read the endpoint from an environment variable, this step was sufficient. However, if the code did not read the endpoint from an environment variable, it revealed a potential bug that could have caused problems in production. The author then ran the test suite, focusing on smoke tests rather than the full suite, to quickly identify any issues with the migration.

During the 25-minute mark, the author compared the outputs from both the local and remote endpoints by running the same prompts against each and diffing the results. This step was crucial in identifying structural differences in the model outputs, such as differences in field names or error shapes. In the author's case, the remote endpoint returned a usage object with token counts, while the local model returned the same shape but with different field names.

This discrepancy caused their monitoring code to silently drop the data, highlighting another potential issue that would have been discovered during the drill.

Finally, in the last 15 minutes, the author rolled back the changes to the free endpoint and documented the findings. The runbook consisted of six steps, detailing the process for migrating between endpoints, testing the application, and checking for any issues. The author concluded that the drill revealed three critical issues: hardcoded URLs in a test fixture, strict JSON parsing that couldn't handle different response shapes, and monitoring code that dropped usage data due to a field name assumption.

By running this drill, the author demonstrated the importance of preparing for potential disruptions when using managed endpoints and emphasized the need to identify and address these potential issues before they cause production incidents.