Why is the connection to S3 via a Gateway Endpoint timing out?

1
votes
VPC:
  PublicSubnet:
    EC2-Instance
    IGW
    NAT
    Route to IGW
  PrivateSubnet:
    EC2-Instance
    Route to NAT
    Route to S3VPCEndpoint
  S3VPCEndpoint

This is the network part of my CFn template:

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      EnableDnsSupport: true
      EnableDnsHostnames: true
      InstanceTenancy: default
      CidrBlock: 10.1.0.0/16

  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      CidrBlock: 10.1.0.0/24
      MapPublicIpOnLaunch: True
      VpcId: !Ref VPC
      Tags:
        - Key: "Name"
          Value: "Raffael Public Subnet"

  PrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      CidrBlock: 10.1.1.0/24
      VpcId: !Ref VPC
      Tags:
        - Key: "Name"
          Value: "Raffael Private Subnet"

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    DependsOn: VPC

  AttachGateway:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC

  PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC

  PublicRouteToIGW:
    Type: AWS::EC2::Route
    DependsOn: AttachGateway
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId:
        Ref: PublicSubnet
      RouteTableId:
        Ref: PublicRouteTable

  PrivateSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId:
        Ref: PrivateSubnet
      RouteTableId:
        Ref: PrivateRouteTable

  PublicInstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH access via port 22
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: 0.0.0.0/0

  PrivateInstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH access via port 22
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          SourceSecurityGroupId: !GetAtt PublicInstanceSecurityGroup.GroupId

  Nat:
     Type: AWS::EC2::NatGateway
     Properties:
        AllocationId: !GetAtt NatEIP.AllocationId
        SubnetId: !Ref PublicSubnet

  NatEIP:
     Type: AWS::EC2::EIP
     Properties:
        Domain: vpc

  PrivateRouteToNat:
     Type: AWS::EC2::Route
     Properties:
        RouteTableId: !Ref PrivateRouteTable
        DestinationCidrBlock: 0.0.0.0/0
        NatGatewayId: !Ref Nat

  S3GatewayEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      RouteTableIds:
        - !Ref PublicRouteTable
        - !Ref PrivateRouteTable
      ServiceName: !Sub com.amazonaws.${AWS::Region}.s3
      VpcId: !Ref VPC

There are two EC2 instances - one in the private and one in the public subnet. These and the relevant Security Groups are set up in a second template.

If I SSH into the private EC2 via the public EC2 I can access S3 from the private subnet. Meaning aws s3 ls lists all buckets.

But - this request is apparently going over the NAT. Because if I effectively deactivate it by setting the DestinationCidrBlock of its route to 1.2.3.4/32 then aws s3 ls is timing out:

HTTPSConnectionPool(host='s3.amazonaws.com', port=443): 
Max retries exceeded with url: / (Caused by ConnectTimeoutError
(<botocore.awsrequest.AWSHTTPSConnection object at 0x7fd86b183828>, 
'Connection to s3.amazonaws.com timed out. (connect timeout=60)'))

So I had a look at Why can’t I connect to an S3 bucket using a gateway VPC endpoint?:

  1. DNS settings in your VPC: "DNS resolution" of the VPC is set to "Enabled"
  2. Route table settings to Amazon S3: There is a route
  3. Security group outbound rules: unrestricted
  4. Network ACL rules: unrestricted
  5. Gateway VPC endpoint policy: unrestricted
  6. S3 bucket policy: I'm testing with ListBucket

Any idea what I have to adjust in my template?

3
amazon-web-servicesamazon-cloudformation
One issue I see is that you are missing required DependsOn attribute on EIP on VPCGatewayAttachment. This is most likely not the reason why S3 gateway is not working, but still good practice to have the attribute. – Marcin
Actually, I am not sure where you are referring to regarding the DependsOn. The link would mean that I have to place DependsOn: AttachGateway into the EC2-Resource (of the public subnet). Is that what you mean? The only EIP here is not related to VPCGatewayAttachment. – Raffael
@Marcin I think you are referring to what is inidicated in the last example here: docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/…. This example confused me b/c there is no VPCGatewayAttach specified and it worked without this DependsOn so I just left it out. – Raffael
Check example in the link you posted. It has DependsOn: VPCGatewayAttach for the EIP. – Marcin

3 Answers

3
votes

Your VPC setup is correct (minus missing DependsOn). I used copy-and-paste to replicate your setup, launched two instances, one public and one private, and everything works as expected.

I also removed the NAT gateway, and still s3 access worked in private subnet. To double check I deleted the S3 VPC gateway, and the access to s3 stopped, indicating that the traffic was going through the gateway.

Thus, there must be something else happening, not related to the VPC setup. Maybe something with your instances? I used us-east-1 region though, thus there maybe is some regional issue in eu-central-1?

0
votes

From docs:

Endpoints currently do not support cross-Region requests—ensure that you create your endpoint in the same Region as your bucket. You can find the location of your bucket by using the Amazon S3 console, or by using the get-bucket-location command. Use a Region-specific Amazon S3 endpoint to access your bucket; for example, mybucket.s3-us-west-2.amazonaws.com. For more information about Region-specific endpoints for Amazon S3, see Amazon Simple Storage Service (S3) in Amazon Web Services General Reference. If you use the AWS CLI to make requests to Amazon S3, set your default Region to the same Region as your bucket, or use the --region parameter in your requests.

Check if this might be the culprit. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints-s3.html

0
votes

From your public subnet EC2 instance:

This EC2 instance as a public IPv4 address and can use an Internet Gateway.

$ aws s3 ls

Should be ok.

$ aws s3 ls --region eu-central-1

Should be ok.

From your private subnet EC2 instance:

This EC2 instance as only private IPv4 address and cannot use the Internet Gateway, you have created a VPCEndpoint for accessing S3.

$ aws s3 ls

Is NOT ok (does not respond).

I guess you probably have S3 buckets in multiple regions?

$ aws s3 ls --region eu-central-1

Should be ok.

You can only list S3 buckets of the "eu-central-1" region because your VPC endpoint is associated with the region "eu-central-1".

You can verify in the AWS console the value created form your CF template:

ServiceName: !Sub com.amazonaws.${AWS::Region}.s3