I Gave an EC2 Instance Zero Internet and It Still Uploaded to S3

The challenge: The client demanded private backend communication with Amazon S3 without direct internet access. Two VPCs were involved: a custom VPC with a private EC2 instance and a default VPC with a public-facing EC2 instance serving as a bastion host. The goal was to execute aws s3 cp from the private EC2 via SSH, maintaining all traffic within the AWS network.

The solution required five components working together: VPC Peering, route tables, security groups, an IAM role, and an S3 Gateway Endpoint. The architecture diagram illustrated the complete design.

Key challenges included:

1. Lack of Internet Gateway and NAT leading to no default route for the private subnet, resulting in S3 requests timing out.

2. Correctly configuring VPC Peering, as traffic only flows when both route tables reference it.

3. Properly securing security groups to prevent cutting off access.

4. Avoiding hardcoding credentials; using IAM roles instead.

5. Deploying an S3 Gateway Endpoint within the private subnet's route table.

Steps included:

1. Requesting and accepting VPC Peering connections.

2. Building symmetric route tables pointing to each other.

3. Tightening security groups to allow only SSH (22) and ICMP from the exact private IP of the other instance.

4. Attaching an IAM role with S3 permissions to the private EC2 instance.

5. Creating an S3 Gateway Endpoint in the private subnet's route table.

Debugging was facilitated by a hop-by-hop map, highlighting checks at each stage - from the laptop to the S3 bucket. Issues such as timeout or 100% packet loss indicated network problems, while AccessDenied errors pointed to IAM role or bucket policy issues.

The outcome was a secure, low-cost design where authentication came from IAM roles, and all traffic to S3 remained within the AWS network, eliminating the need for static credentials and reducing the attack surface.