EC2 cannot access aws s3 even it has IAM rule to do that

Question

In a VPC, I have two Subnets, one is a public subnet with an EC2 instance, the other is a private subnet with 2 EC2 instances. All 3 EC2 instances have the same IAM role to access S3.

The EC2 instance in the public subnet can access S3 directly if I login and run aws s3 ls. However, both of the EC2 instances in the private subnet cannot. What can be the reasons?

EC2 in the private subnet uses a Security Group that accepts traffic from the whole VPC.

EC2 in the public subnet use a Security Group that accepts traffic from Anywhere.

All 3 EC2 instances use the same routing table, same NACLs, use the same IAM role, with the policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

If I manually create a credential profile in the EC2 in the private Subnet, then it can login and access "aws s3 ls".

Update: The routing table of the private subnets does have a VPC Endpoint. The routing table is:

dest=10.1.0.0/16 (myVPC)->target=local
dest=0.0.0.0/0, target=iGW
dest=pl-68a54001 (com.amazonaws.us-west-2.s3)->target=vpce-26cf344f

Among them, #3 means the EC2 can access S3 via VPC endpoint. #2 is added because an ELB is in front of the EC2 instance and has to access internet.

Another observation: If I enable Assign Public IP in the private subnet and then launch a new EC2 instance, this EC2 instance can access S3. If I disable Assign Public IP in the private subnet and then launch a new EC2 instance, the new EC2 instance cannot access S3.

BTW, I already have region set as us-west-2 before running terraform:

[ec2-user@ip-XXXXX]$ echo $AWS_DEFAULT_PROFILE
abcdefg
[ec2-user@XXXXX]$ aws configure --profile abcdefg
AWS Access Key ID [****************TRM]: 
AWS Secret Access Key [****************nt+]: 
Default region name [us-west-2]: 
Default output format [None]:

John Rotenstein John Rotenstein · Accepted Answer · 2017-09-03T21:36:23

The fact that your solution works when an instance has a Public IP address, but does not work when it does not have a Public IP address, suggests that the instance is actually in a public subnet.

Indeed, looking at "the routing table of the private subnets", you include this line:

2. dest=0.0.0.0/0, target=iGW

This is making the subnet a Public Subnet because it is pointing to an Internet Gateway.

To clarify:

A public subnet is a subnet with a Route Table entry that points to an Internet Gateway
A private subnet is a subnet with a Route Table that does not point to an Internet Gateway

Therefore, you should remove the above entry from the Route Table for your private subnets if you actually want them to be private.

Next, it appears that you aws s3 ls request is not being sent to the VPC Endpoint. This might be because you are not sending traffic to s3.us-west-2.amazonaws.com as listed in your route table. Try this command:

aws s3 ls --region us-west-2

That will send the request to the S3 endpoint that would be routed via the VPC Endpoint. You should direct all of your S3 commands to that region, since the VPC Endpoint only points to the region in which it was configured.

EC2 cannot access aws s3 even it has IAM rule to do that

2 Answers