EC2 Instance cannot connect to ECS Cluster

1
votes

Helo, I have empty AWS ECS Cluster but I am unable to put instances into it. I wanted to use Launch templates and Autoscaling Group, but I am unable to assign created EC2 Instance.

The issue is in shown in ecs-agent.log

level=error time=2020-10-17T23:23:37Z msg="Unable to register as a container instance with ECS: RequestError: send request failed\ncaused by: Post \"https://ecs.eu-central-1.amazonaws.com/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" module=client.go
level=error time=2020-10-17T23:23:37Z msg="Error registering: RequestError: send request failed\ncaused by: Post \"https://ecs.eu-central-1.amazonaws.com/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" module=agent.go

Notes:

  • Using AMI ami-0eff571a24849e852
  • Cluster name: debug
  • Region is eu-central-1
  • Instance has no public IP
  • Instance is in 10.10.100.0/24 subnet (10.10.100.14) and VPN subnet is 10.10.0.0/16
  • Instance can reach the internet through NAT Instance:
[ec2-user@ip-10-10-100-14 ecs]$ ping google.com
PING google.com (216.58.212.142) 56(84) bytes of data.
64 bytes from ams15s21-in-f14.1e100.net (216.58.212.142): icmp_seq=1 ttl=109 time=50.1 ms
64 bytes from ams15s21-in-f142.1e100.net (216.58.212.142): icmp_seq=2 ttl=109 time=40.1 ms
  • DNS to outside is resolving fine
[ec2-user@ip-10-10-100-14 ecs]$ nslookup google.com
Server:     10.10.0.2
Address:    10.10.0.2#53

Non-authoritative answer:
Name:   google.com
Address: 216.58.212.142
  • Just to be sure, I have created Endpoints from VPC and Subnet where Instance is to ECS VPC-Endpoints
  • I have attached the security group with no restrictions for test security-group
  • ecs.config:
ECS_CLUSTER=debug
ECS_BACKEND_HOST=

[ec2-user@ip-10-10-100-14 ecs]$ nslookup ecs.eu-central-1.amazonaws.com
Server:     10.10.0.2
Address:    10.10.0.2#53

Non-authoritative answer:
Name:   ecs.eu-central-1.amazonaws.com
Address: 10.10.100.219

Does anyone have any suggestions?

1
amazon-web-servicesamazon-ec2
Your ECS cluster is called debug? Also can you go to ECS Console and look into Events what does it report? - Marcin
@Marcin yes, cluster name is debug... for debug :) Where can I see Events? I have no services, so I believe there will be no events for now. - Filip Niko
Have you inspected /var/log/ecs/ecs-agent.log? - Marcin
@Marcin sure, you can find the output on the beginning of my question - Filip Niko
The security groups you've posted are for instance or your VPC interface endpoints? - Marcin

1 Answers

1
votes

I have solved this issue.

The problem was caused by VPC Endpoints. I placed them in the wrong subnet and security group.

I have moved them to the public subnet (next to NAT Instance) and in the same group as NAT Instance (allow incoming traffic from sg where my EC2 is).

According to docs:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/vpc-endpoints.html

The security group attached to the VPC endpoint must allow incoming connections on port 443 from the private subnet of the VPC.