I Accidentally Deleted Our Production S3 Bucket — Here's What Happened in the Next 4 Hours

In the span of four hours, a cloud engineer accidentally deleted a production S3 bucket, causing significant data loss. The engineer had mistakenly run the aws s3 rb command in the wrong terminal window, believing they were deleting a staging bucket when they actually deleted the production bucket containing three years of user-uploaded files. This led to the immediate loss of user profile pictures, document uploads, exported reports, and some application assets.

Fortunately, the engineer had Versioning enabled on the production bucket, which prevented the deletion of individual objects. However, the deletion of the bucket itself removed all versions and objects. The engineer then discovered Cross-Region Replication (CRR) had been set up eight months earlier as a compliance requirement, but it had never been tested. CRR continuously replicated objects from the source bucket to a destination bucket in another region, and the destination bucket still contained all 47,832 objects.

The recovery process involved creating a new source bucket, copying the objects from the replica bucket to the new source bucket, updating bucket policies, CORS configuration, and lifecycle rules, and re-enabling CRR. This process took approximately 2 hours and 47 minutes, restoring all the lost data. While some bucket policy and lifecycle rule permissions were missing, the engineer was able to manually recreate them.

The incident highlighted the importance of versioning, cross-region replication, and treating infrastructure like code, by version-controlling policies, lifecycle rules, and CORS configurations.