{
  "id": 18705536,
  "title": "The 2017 S3 Outage Was One Wrong Input. AWS Fixed the Tool, Not the Engineer",
  "url": "https://urgent.news/2026/11/07/the-2017-s3-outage-was-one-wrong-input-aws-fixed-the-tool-not-the",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-07T12:01:27.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/gabrielanhaia/the-2017-s3-outage-was-one-wrong-input-aws-fixed-the-tool-not-the-engineer-5a7b"
  },
  "original_language": "en",
  "account": "On February 28, 2017, a member of the Amazon S3 team executed a command from a standard procedure. The intention was to take a few servers out of one subsystem in the us-east-1 region. However, an incorrectly entered input led to a larger number of servers being removed than intended. S3 in that region remained non-operational for over four hours, from 9:37 AM Pacific to 1:54 PM, due to this one input error. AWS resolved the issue by updating the tool, rather than addressing any engineering shortcomings.\n\nThe S3 team was troubleshooting the S3 billing system, which was performing slower than expected. The command was designed to remove a small set of servers from a subsystem used by the billing process. The incorrect input instead removed a significantly larger number of servers, affecting two other subsystems: the index subsystem, which manages metadata and location of every S3 object, and the placement subsystem, which allocates storage for new objects. Both subsystems required a full restart.\n\nAWS admitted that they had not completely restarted the index subsystem or the placement subsystem in larger regions for many years. The restart and safety checks on the metadata took longer than expected due to years of growth. After the incident, the index could serve GET, LIST, and DELETE requests again by 12:26 PM, and was fully recovered by 1:18 PM. The placement subsystem finished its recovery at 1:54 PM, bringing PUT requests back online.\n\nThe incident impacted several services, including the S3 console, new EC2 instance launches, EBS volumes that required data from an S3 snapshot, and AWS Lambda. The AWS Service Health Dashboard could not display individual service statuses until 11:37 AM, as its admin console depended on S3.\n\nAWS implemented changes to the tool to prevent similar events in the future. The tool now removes capacity more slowly and refuses to cross a minimum capacity floor, preventing an incorrect input from triggering a similar event. AWS also audited other operational tools to ensure similar safety checks were in place. The changes were focused on the tool itself, not on the person who entered the incorrect input. AWS moved to split the index subsystem into smaller cells and moved the Service Health Dashboard's admin console to run across multiple regions.",
  "summary": "Book: System Design Pocket Guide: Fundamentals Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub At 9:37 AM Pacific on February 28, 2017, an authorized member of the Amazon S3 team ran a command from an…",
  "key_points": [
    "S3 team executed command with incorrect input, removing more servers than intended",
    "Incident caused four-hour outage from 9:37 AM to 1:54 PM Pacific",
    "AWS fixed the tool, not the engineer, implementing safety changes to prevent recurrence"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}