The 2017 S3 Outage Was One Wrong Input. AWS Fixed the Tool, Not the Engineer

On February 28, 2017, a member of the Amazon S3 team executed a command from a standard procedure. The intention was to take a few servers out of one subsystem in the us-east-1 region. However, an incorrectly entered input led to a larger number of servers being removed than intended. S3 in that region remained non-operational for over four hours, from 9:37 AM Pacific to 1:54 PM, due to this one input error. AWS resolved the issue by updating the tool, rather than addressing any engineering shortcomings.

The S3 team was troubleshooting the S3 billing system, which was performing slower than expected. The command was designed to remove a small set of servers from a subsystem used by the billing process. The incorrect input instead removed a significantly larger number of servers, affecting two other subsystems: the index subsystem, which manages metadata and location of every S3 object, and the placement subsystem, which allocates storage for new objects. Both subsystems required a full restart.

AWS admitted that they had not completely restarted the index subsystem or the placement subsystem in larger regions for many years. The restart and safety checks on the metadata took longer than expected due to years of growth. After the incident, the index could serve GET, LIST, and DELETE requests again by 12:26 PM, and was fully recovered by 1:18 PM. The placement subsystem finished its recovery at 1:54 PM, bringing PUT requests back online.

The incident impacted several services, including the S3 console, new EC2 instance launches, EBS volumes that required data from an S3 snapshot, and AWS Lambda. The AWS Service Health Dashboard could not display individual service statuses until 11:37 AM, as its admin console depended on S3.

AWS implemented changes to the tool to prevent similar events in the future. The tool now removes capacity more slowly and refuses to cross a minimum capacity floor, preventing an incorrect input from triggering a similar event. AWS also audited other operational tools to ensure similar safety checks were in place. The changes were focused on the tool itself, not on the person who entered the incorrect input.

AWS moved to split the index subsystem into smaller cells and moved the Service Health Dashboard's admin console to run across multiple regions.