{
  "id": 11376838,
  "title": "Our bucket policy let the office in and locked out every server we own",
  "url": "https://urgent.news/2026/10/02/our-bucket-policy-let-the-office-in-and-locked-out-every-server-we-own",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-02T06:43:22.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sergey_shinder_ab2d943365/our-bucket-policy-let-the-office-in-and-locked-out-every-server-we-own-49eg"
  },
  "original_language": "en",
  "account": "Our security team requested in June that the bucket containing signed contracts be accessible only from the office and our own network. I created the bucket policy in Terraform, ensuring that any S3 action would only be allowed if the request's aws:SourceIp matched the office's range or the NAT gateway's public addresses. From my office laptop and a private subnet instance, both test runs were successful. The plan was reviewed and approved on a Wednesday. However, by Friday morning, the contracts service experienced access denied errors for both uploads and downloads due to an explicit deny.\n\nDuring the evening of Thursday, the network team added a gateway endpoint for S3 to the VPC to avoid paying NAT charges for unnecessary traffic. While this was a positive change, it inadvertently caused issues. Now, requests to S3 from within the VPC no longer passed through the NAT. Instead, they traveled through the endpoint, which only held public addresses. Consequently, none of our servers' requests carried an address on my list, causing the condition to no longer match for any of them. The deny applied to all requests, making it harder to undo the policy changes. The deny covered s3:*, including modifying the bucket policy, and our CI runners were located in the same VPC. Terraform could not remove the policy it had written, requiring the use of the account root user to delete the policy, which took forty minutes to locate the individual holding the second factor. The policy now permits the VPC through aws:SourceVpce, naming the endpoint, and the office through aws:SourceIp. The deny only applies when a request matches neither condition. A break glass role is excluded by aws:PrincipalArn, eliminating the need for root access in the future. Any policy change containing a network condition is now run in the pipeline through the IAM policy simulator, using the endpoint and office as request contexts. Additionally, the endpoint, route tables, and policies affected by them are now housed in a single module, ensuring that a network change will appear in the same plan as everything it impacts. The condition on where a request originates has been updated to reflect the network's current state. Ours was reconfigured the following day by Sergey Shinder.",
  "summary": "In June our security team asked for the bucket holding signed contracts to be reachable only from the office and from our own network. I wrote the bucket policy in Terraform: deny every S3 action unless the request's aws:SourceIp is one of the office ranges or one of the two public addresses of our NAT gateways. I tested it from my laptop in the office and from an instance in a private subnet,…",
  "key_points": [
    "Bucket policy allowed office and network access only",
    "Network team added gateway endpoint for S3 traffic",
    "Deny rule blocked all requests, causing access errors"
  ],
  "editors_take": "The incident highlights the need for integrated planning and testing of network and security policy changes to avoid inadvertently blocking access and requiring emergency intervention with elevated privileges.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}