{
  "id": 2346136,
  "title": "The System at Night",
  "url": "https://urgent.news/2026/08/21/the-system-at-night",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-21T09:17:51.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/frederikvonderheyden/the-system-at-night-36h7"
  },
  "original_language": "en",
  "account": "Waking up to automated incident reports at 3:47 AM was an eye-opening experience. My system had detected an unexpected behavior during an overnight automation run, traced the root cause to a malformed config value, applied the fix, ran a verification pass, and wrote the entire incident to my knowledge vault. The timestamp, root cause, solution, and follow-up task were all documented before I even made coffee. This system learns to handle such situations independently, without requiring constant supervision.\n\nThe key difference between a script and a system is that a script does exactly what you tell it, while a system does what you would tell it, even when you're not there. Most developers stop at writing scripts, creating 40 independent processes that don't communicate with each other and require constant human intervention when unexpected issues arise. However, my system operates differently, with 232 automated processes running continuously, sharing context and writing to a central knowledge base. When an unexpected issue occurs, the system attempts to resolve it, documents the findings, and flags the incident for review in a structured format that can be processed efficiently the following day.\n\nThe knowledge vault consists of 17,812 files, not manually written documentation, but the accumulated output of every process that has run within the system. Each unexpected outcome becomes a file, and every resolved issue becomes a reusable pattern. The incident files follow a strict template, including the incident name, timestamp, process, root cause, fix applied, verification status, and follow-up tasks. Over time, the patterns directory grows into a genuine institutional memory, allowing the system to find relevant patterns and apply them to similar issues in the future.\n\nThe error detection mechanism is a simplified bash script that wraps around most long-running processes. The `run_with_capture` function captures the output of the process, checks for errors, and, if an error occurs, generates an incident file in the vault with a strict format: timestamp, process name, timestamp, exit code, output, and pending fix applied and follow-up tasks. If known fixes are available, the `apply_known_fix` function looks up matching signatures in the patterns directory, providing a dumb, reliable, and increasingly smart lookup table.\n\nInitially, my system required constant human intervention, handling every edge case and queuing unexpected results. The problem was not that automation broke, but that most automation architectures were not designed to handle breakage gracefully. They assumed happy paths and failed silently when things went wrong. The turning point was a cascade failure in early 2024, where three interdependent processes failed overnight. The lack of context and a trail to follow made it difficult to reconstruct what went wrong, leading to a day spent reconstructing the failures from scattered logs.\n\nTo prevent such issues, I rebuilt the system around one principle: every process must leave a trail that a future version of me or an automated process can follow without additional context. This meant structured output, consistent file naming, a queryable knowledge vault, and error handling that automatically writes to the vault, regardless of success or failure. The implementation cost is relatively low, with a Linux server (around 6 EUR per month), bash scripts with structured output, a flat-file vault, and a daily summary job aggregating overnight activity. The daily summary script runs at 6 AM and sends a single message containing all incidents logged overnight, typically no more than three lines or twenty lines, providing a clear overview of what occurred during the night.",
  "summary": "My System Fixed a Bug at 3:47 AM While I Was Asleep I woke up to a notification. Not an alert. Not a page. A report. My system had detected an unexpected behavior during an overnight automation run, traced it to a malformed config value, applied a fix, ran a verification pass, and written the entire incident to my knowledge vault. Timestamp, root cause, solution, follow-up task. All before I made…",
  "key_points": [
    "System autonomously detects and resolves issues overnight",
    "232 automated processes run continuously with shared context",
    "Knowledge vault contains 17,812 files from system's operations"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}