AWS ECR, ECS based deployment: nginx: [emerg] host not found in upstream “app:9000” in /etc/nginx/conf.d/default.conf:2

Question

I am struggling to configure AWS ECS Task definition to run simple PHP-FPM and Nginx based containers.

My "app" container is running at app:9000 port and Nginx is in 80 port. Nginx is forwarding the request to app container through a fastcgi_pass app-upstream;

All of these are running perfectly in local. The same configuration is running perfectly in DigitalOcean Docker Instance but fails in AWS ECS.

I am guessing some task-definition configuration issue, but I can't find it.

Error Logs:

I am getting this log from Nginx container

nginx: [emerg] host not found in upstream "app:9000" in /etc/nginx/conf.d/default.conf:2

and this log from App (PHP-FPM) container

ECS

I've created a simple cluster with T2 Small instance, which is running perfectly.

Dockerfile

In this Github Repo, I've added the Dockerfile for App, Nginx image, docker-compose.yml file and task-defination.json file for reference if there is any mistake in docker files.

Source code in Github repo: https://github.com/arifulhb/docker-ecr-sample

Answer 1

Your issue is related to wrong upstream path mentioned in nginx configuration.

Do following things to investigate, also avoid using custom container names in docker-compose files if specifically not needed :-

Quick resolution would be :-

1. Remove container names from docker-compose files.
2. The service key name ( for example : test_app ) mentioned in docker-compose file is treated as container name automatically so use that.
3. The correct upstream path after making above changes should be test_app:9000

Proper and recommended way of building a docker-compose files :-

1. Create a custom docker network suppose with name "intranet"
2. Mention this network "intranet" in each service you create in your docker-compose file.
3. Follow steps mentioned in quick resolution

How does this helps you ? You have the ability to inspect this network you created, figure out if your containers are properly connected and identify the names used for connection.

Command : docker network inspect <network_name>

NOTE : Docker treats container names as host names by default for internal connections.

Answer 2

When using multiple container, the container name is very important to provide internal connectivity.

As I see your Docker compose file, the container name should match the name used in nginx conf.

version: '3'
services:
  test_app:
      container_name: app # not test_app_1
      volumes:
        - test-app-data:/var/www/app
  test_nginx:
      image: "xxxxxx.dkr.ecr.xx-xx-1.amazonaws.com/test-nginx"
      build:
        context: ./docker/nginx
        dockerfile: Dockerfile
      container_name: nginx
      ports:
        - "80:80"
        - "443:443"
      volumes:
        - test-app-data:/var/www/app
        - test-nginx-log:/var/log/nginx
      external_links:
        - app # not test_app_1
      depends_on:
        - test_app
volumes:
  test-app-data:
  test-nginx-log:

Answer 3

I got the same problem too. My "blog" container is running at blog:8000 port and "nginx" container is at 80 port. "nginx" container is forwarding the request to "blog" container.

Because I didn't set "Links" in "NETWORK SETTINGS" for "nginx" container at all.

So I put the name of the back container "blog" in the front container "nginx"("Links" in "NETWORK SETTINGS" for "nginx" container).

Then, it was successful to run both "blog" and "nginx" containers properly.

So in your case, put the name of the back container "app" in the front container "nginx"("Links" in "NETWORK SETTINGS" for "nginx" container). It will work. I used Adiii's solution.

Don't forget to put "CMD-SHELL, curl -f http://localhost:9000/ || exit 1" in "Command" in "HEALTHCHECK" for "app" container.

Moreover, Don't forget to put "app" in "Container name" and select "HEALTHY" in "Condition" in "STARTUP DEPENDENCY ORDERING" for "nginx" container.