{
  "id": 4882797,
  "title": "From 3:00 AM Panic to Confidence: How I Use AI During On-Call Incidents",
  "url": "https://urgent.news/2026/09/01/from-3-00-am-panic-to-confidence-how-i-use-ai-during-on-call-incidents",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-01T15:13:33.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/qainsights/from-300-am-panic-to-confidence-how-i-use-ai-during-on-call-incidents-2nk4"
  },
  "original_language": "en",
  "account": "In the early hours of a night, a PagerDuty alert begins to buzz. A developer's eyes flutter open, opening an incoming alert to discover mounting error rates, active Slack channels, and an incensed incident commander waiting for an update. This scenario is familiar to many developers who have been on call. The first few minutes are spent feeling panicked, trying to decide which dashboard to open and how to explain the issue while still investigating. However, AI can help to ease the early morning panic. It does not eliminate the panic factor or replace the engineer, but it can remove the first few minutes of searching, tab switching, and gathering context.\n\nThe author started with a small prompt in their notes folder. As they opened an alert, they filled in the blanks and started the investigation. The prompt was simple: \"I received this alert: PagerDuty or Slack alert link, context, error or symptom, investigation window, runbook.\" It guided them to analyze relevant Splunk logs and dashboards, check recent deployments, configuration changes, feature-flag changes, upstream and downstream dependencies, cloud-provider status pages, internal maintenance announcements, and PagerDuty history. The output was divided into five sections: confirmed facts, leading hypotheses, unknowns and missing evidence, customer impact, and recommended next checks.\n\nThis prompt was not meant to resolve the incident by itself, but to get the investigation moving while the engineer opened the runbook, inspected main dashboards, and applied their knowledge of the service. AI excels at gathering and correlating information across tools, but it is still the engineer's responsibility to understand whether the evidence makes sense for the system. The prompt was tested before needing it in production by replaying previous incidents whose root causes were already known. This step was crucial to ensure the AI did not confuse correlation for causation and stayed within the correct environment and region.\n\nAs skills evolved, the author converted the prompt into an /investigate skill. This skill collects the required context before launching the investigation, asking for missing required context such as service name, environment, region, alert or incident link, symptom category, approximate start time and investigation window, known or potential customer impact, and runbook link. This skill ensures that the investigation starts with accurate and complete information, saving precious time during an on-call incident.",
  "summary": "In this blog post, we will see how I use AI to speed up incident investigation without letting it take over the decisions that need a human. It is 3:00 AM. Your phone starts making that familiar PagerDuty noise. You open the alert with half-open eyes. Error rates are climbing. Slack is already active. The incident commander wants an update. Depending on the severity, your director or CTO may also…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}