{
  "id": 10387473,
  "title": "A mistake changed my career — production went down at 2am.",
  "url": "https://urgent.news/2026/09/28/a-mistake-changed-my-career-production-went-down-at-2am",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-28T07:01:56.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/infoinlet1/a-mistake-changed-my-career-production-went-down-at-2am-13j"
  },
  "original_language": "en",
  "account": "On a dark and quiet night, a developer awoke to a phone call. A customer was locked out of their own account, and the developer knew something was amiss. The system had reported everything as perfect, but the customer's experience told a different story. This was a turning point for the developer, who realized that the system's \"success\" message was not always trustworthy.\n\nThe developer traced the issue and found a simple bug. The system acknowledged the request before saving the data, causing customers to be locked out during a brief window between the acknowledgment and the actual persistence of the data. This flaw had gone unnoticed during testing and review, as the system's \"success\" message was emitted before the actual success could be verified.\n\nThe developer learned a valuable lesson: the system that writes the code should not be the same system that swears it works. The person who writes the code should be separate from the person who verifies its correctness. This separation ensures that the code is not only clean and well-reviewed but also rigorously tested under various scenarios, including edge cases and potential failures.\n\nThe developer now follows the principle of \"ack-after-persist.\" The system only emits a \"success\" message after the data has been successfully persisted. This simple change ensures that the system's witness to its own success aligns with the reality of the system's state.\n\nThe developer's experience taught them that AI, like any other tool, can be misled by the same blind spots as a human. AI may produce clean, confident, and plausible output, but it should not be trusted to certify its own success. The developer now treats the author's (human or machine) \"success\" claim as a truth claim that needs to be verified.\n\nTo test the robustness of their systems, the developer recommends asking critical questions, such as: What happens under a retry? What if the input is hostile? What if the network connection is unstable? By proactively breaking the system and testing it in various scenarios, developers can build confidence in their code and avoid relying solely on the system's self-acknowledgment of success.\n\nIn the age of AI-assisted development, the developer's lesson is more relevant than ever. AI can write code that appears perfect, but it too can be fooled by its own blind spots. The developer's approach of making the \"author\" a separate witness from the \"witness\" ensures that the system's code is not only well-written but also thoroughly tested and reliable, regardless of whether it was written by a human or an AI.",
  "summary": "The worst bug of my career didn't wake me with an error. It woke me with a phone call. 2am. A paying customer, locked out of their own account, more confused than angry — which was worse. They'd done the thing. They'd seen it go through. And now the door was shut and there was no record they'd ever knocked. I did what you do. I opened the logs, hands already moving, ready to grep for the red…",
  "key_points": [
    "Developer's system success message was not always trustworthy",
    "Bug caused customers to be locked out during brief window",
    "Developer learned importance of separating code writer from verifier"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}