When your AI system breaks—and what to do about it

Every AI tool comes with the caveat that it may produce mistakes. A workflow that operates flawlessly one day may not function properly the following day. This is a constant challenge. Unlike conventional software, an AI system that seems to be operating smoothly needs periodic monitoring or adjustments. AI can fail in predictable ways.

For example, it might be linked to an external tool like your email, and the connection might have expired. Some AI tools require your computer to remain on or awake for tasks to be completed. You may receive an error message or no output at all. These failures are relatively easy to resolve, as the problem is often clear. You reconnect and attempt the task again.

I depend significantly on AI scheduled tasks and have a notification system in place. If anything goes wrong, I receive an email alerting me to review the process. This allows me to promptly identify any issues and address them. The unpredictable AI failures The most challenging failure occurs when an AI process runs as intended, but the output becomes inferior.

AI models and features change frequently, often daily. A prompt that yielded the desired results last month may produce noticeably different outcomes today. Professionals in the field refer to this phenomenon as "drift." Most setups lack a mechanism to detect this drift, since technically nothing has broken. I have used Claude and encountered difficulties with Opus 5.0.

For how I utilize AI, the output was "off" in an way I couldn't clearly articulate. I found myself reverting to Opus 4.6, which was the last model I felt comfortable with. Interestingly, the transition to Opus 5.5 was much smoother. The most reliable method to identify drift is to consistently review AI output, regardless of how long the system has been functioning as expected.

An occasional review the same critical eye you initially gave it when testing an AI process. Drift is inconvenient, particularly when the purpose of automation is to save you time. However, you cannot expect a model to maintain the same performance from one month to the next. I view it this way: dedicating a mere 5 minutes to review is still more efficient than spending 20-30 minutes performing the task manually.

When incorporating AI into your routine, it is essential to account for the time required for AI maintenance. Some fixes require immediate attention. When a notification informs me that something has failed, I often need to address it promptly. Depending on the nature of the issue, it could disrupt my entire workflow until the problem is resolved.

Other changes necessitate setting aside time for thorough review. When a new model is announced, I do not immediately switch to the new version. A significant change in output could disrupt an afternoon, so I allocate time to test it first. Upkeep is an inherent cost associated with any AI system you deploy. Moreover, if a process consumes more time to maintain than it saves you, it is justifiable to question its continued use.