"Network bindings - not configured" when running service with AWS Fargate

5
votes

I'm trying to set up a couple of services with ECS Fargate, provisioned via Terraform. They use the same module, only image, ALB target group, environment variables and port mappings differ.

2 out 3 services start their tasks successfully only one (unfortunately the main service), doesn't want to start and shows Network bindings - not configured for the container. The port I'm using is 80.

The task definition has the correct port mappings.

I've tried changing the port (to 8080), use multiple port mappings and recreating the service multiple times to no effect.

Of course the task gets killed by the load balancer for failing health checks.

Any pointers what could be wrong? I found some Github issues regarding this from 2017, but on EC2-backed ECS instances, which has been claimed to be fixed.

For reference, here's the task definition JSON:

{
  "ipcMode": null,
  "executionRoleArn": "ROLE_ARN",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/drone",
          "awslogs-region": "eu-central-1",
          "awslogs-stream-prefix": "drone-server/"
        }
      },
      "entryPoint": null,
      "portMappings": [
        {
          "hostPort": 80,
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "command": null,
      "linuxParameters": null,
      "cpu": 256,
      "environment": [...],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": [...],
      "dockerSecurityOptions": null,
      "memory": 512,
      "memoryReservation": 512,
      "volumesFrom": [],
      "stopTimeout": 30,
      "image": "drone/drone:1",
      "startTimeout": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": false,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "drone-server"
    }
  ],
  "placementConstraints": [],
  "memory": "512",
  "taskRoleArn": "ROLE_ARN",
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "TASK_DEFINITION_ARN",
  "family": "drone-server",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.container-ordering"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.secrets.ssm.environment-variables"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "256",
  "revision": 14,
  "status": "ACTIVE",
  "proxyConfiguration": null,
  "volumes": []
}```
2
amazon-web-servicesnetworkingamazon-ecsaws-fargate
To be clear, are you running 3 services on the same task? Or do you have three separate ECS services and only two of them start? – Brett Green
There are 3 services with almost identical, but slightly different, task definitions having one container each. – Markus Mühlberger

2 Answers

2
votes

With ECS on EC2, your container port (like 80) is mapped to a dynamic port on the host (like 35467) and then registers this port with the TargetGroup with type 'instance'. (Technically, this happens if you send a zero as the host port mapped to port 80 on the container. AWS takes this as 'dynamically assign a port on the host')

The big difference in Fargate is it uses ENIs attached to task for networking and each task gets its own private IP address (can be public if you want as well).

Then, with that unique IP address (as opposed to instance-unique port) it registers the unique IP address with port 80 to the TargetGroup with type 'ip'.

So two things could be going wrong... first of all, on Fargate, your task must have the same host port and container port (e.g. 80:80), and you must be sure it's registering to the TargetGroup with type 'ip'.

I am not a terraform user, so not sure how much of that is in your control, but I suspect one of those two things is not right and causing your web service/task to not launch correctly.

For reference, here's the task definition JSON:

{
  "ipcMode": null,
  "executionRoleArn": "ROLE_ARN",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/drone",
          "awslogs-region": "eu-central-1",
          "awslogs-stream-prefix": "drone-server/"
        }
      },
      "entryPoint": null,
      "portMappings": [
        {
          "hostPort": 80,
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "command": null,
      "linuxParameters": null,
      "cpu": 256,
      "environment": [...],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": [...],
      "dockerSecurityOptions": null,
      "memory": 512,
      "memoryReservation": 512,
      "volumesFrom": [],
      "stopTimeout": 30,
      "image": "drone/drone:1",
      "startTimeout": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": false,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "drone-server"
    }
  ],
  "placementConstraints": [],
  "memory": "512",
  "taskRoleArn": "ROLE_ARN",
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "TASK_DEFINITION_ARN",
  "family": "drone-server",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.container-ordering"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.secrets.ssm.environment-variables"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "256",
  "revision": 14,
  "status": "ACTIVE",
  "proxyConfiguration": null,
  "volumes": []
}```
0
votes

Apparently Fargate is not very good at reporting errors or displaying state. It doesn't show all the environment variables or the correct status in the AWS console, but somehow works anyways.

Morale of the story is, if something doesn't show up in the console, make sure to test if it does actually not work.

I honestly can't tell a solution to my issues, since when I turned on trace logging on the Drone CI server via an environment variable, it went away.