AWS Fargate - CannotPullContainerError (500)?

Question

I'm using AWS ECS to host my services. When I try to define task with fargate, I'm getting this below problem.

CannotPullContainerError: API error (500): Get https://xxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Further I gave full permissions to access ECR in the IAM user as well. Please help me to sort out this problem.

Answer 1

Have a look here: https://github.com/aws/amazon-ecs-agent/issues/1128

In particular, the comment by samuelkarp

The error you are seeing below is commonly due to lack of internet access to pull the image. The image pull occurs over the network interface used by the Task, and as such shares security group and routing rules.

Please check your configuration for the following:

If you are launching a task without a public IP, make sure that the route table on the subnet has "0.0.0.0/0" going to a NAT Gateway or NAT instance to ensure access to the internet. If your route table has an internet gateway, this is acting like a firewall and preventing the connection from being made. If you are launching a task with a public IP, make sure that the route table on the subnet has "0.0.0.0/0" going to an internet gateway to ensure you will be able to use the public IP successfully for ingress traffic. Verify your security group rules for the Task allows for outbound access. The default here is typically All Traffic to 0.0.0.0/0. If neither of those networking changes apply to you or if they do not fix your problem, please let us know so we can further assist.

Answer 2

I've already answered this here, but copy-paste does not hurt.

The specification for creating a working NAT Gateway is lacking. At the GitHub issue Amazon technicians keep repeating you "just" need Private IP + NAT, however this is not true. I struggled with this myself a lot, but finally got it working properly without using a Public IP for my Fargate services.

To have Fargate services access internet without having a Public IP you need to set up a VPC which has 2 subnets:

• A public subnet with an Internet Gateway allowing bidirectional internet access
• A private subnet with a NAT Gateway allowing only outgoing internet access

You can create such a VPC in 2 ways: by going to Services > VPC > VPC Dashboard, clicking on Launch VPC Wizard and selecting "VPC with Public and Private Subnets"; or manually:

NOTE: All of the following steps are performed in Services > VPC

1. Go to Your VPCs and Create a VPC
2. Go to Subnets and Create subnet 2 times
   1. private subnet
      1. Attach it to the VPC in focus. Whatever CIDR block, whatever availability zone you like
   2. public subnet
      1. Attach it to the VPC in focus. Whatever CIDR block, whatever availability zone you like
3. Go to Internet Gateways and Create internet gateway
   1. Name it however you want
   2. Select the newly created Internet Gateway, Actions, Attach to VPC and attach it to the VPC in focus
4. Go to NAT Gateways and Create NAT Gateway
   1. Important: Select the public subnet
   2. Create New EIP or use an existing one given that you have one
   3. Wait for the gateway to become Available
5. Go to Route Tables and Create route table 2 times
   1. private route table
      1. Attach it to the VPC in focus
      2. Back at the list, select the route table
      3. Routes tab on the bottom, Edit routes
      4. Add route, destination: 0.0.0.0/0, target the NAT Gateway created previously and Save routes
      5. Still having the route table selected, Actions and Set Main Route Table (if not already)
   2. public route table
      1. Attach it to the VPC in focus
      2. Back at the list, select the route table
      3. Routes tab on the bottom, Edit routes
      4. Add route, destination: 0.0.0.0/0, target the Internet Gateway created previously and Save routes
      5. Subnet Associations tab on the bottom, Edit subnet associations
      6. Select the public subnet, Save
6. Put cucumber on eyes.

Every service you put in the public subnet will have bidirectional internet access and every service you put in the private subnet will have only outgoing internet access (yes, Fargate and EC2 services in the private subnet without Public IPs will have internet access).