Connection refused while connection to upstream - Docker

Question

Error that I'm getting:

nginx_prod_vet | 2019/03/07 20:57:11 [error] 6#6: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: , request: "GET /backend HTTP/1.1", upstream: "http://172.23.0.2:81/backend", host: "localhost:90"

My goal is use nginx as reverse-proxy to delivery the frontend files and proxy the other services to the frontend, so it would be accessible localhost:90/backend been call from localhost:90/.

I tried to access from outside the container the backend but it gives me the error above.

Here are the most relevant files:

# docker-compose.yml

version: '3'

services:

  nginx:
    container_name: nginx_prod_vet
    build:
      context: .
      dockerfile: nginx/prod/Dockerfile
    ports:
      - "90:80"
    volumes:
      - ./nginx/prod/prod.conf:/etc/nginx/nginx.conf:ro
    networks:
      - main
    depends_on:
      - backend

  backend:
    container_name: backend_prod_vet
    build:
        context: .
        dockerfile: apache/Dockerfile
    ports:
      - "81:81"
    networks:
      - main

networks:
  main:
    driver: bridge

---

# apache/Dockerfile
FROM httpd:2.4.32-alpine

RUN apk update; \
    apk upgrade;

# Copy apache vhost file to proxy php requests to php-fpm container
COPY apache/apache.conf /usr/local/apache2/conf/apache.conf
RUN echo "Include /usr/local/apache2/conf/apache.conf" \
>> /usr/local/apache2/conf/httpd.conf

---

# apache/apache.conf
ServerName localhost

LoadModule deflate_module /usr/local/apache2/modules/mod_deflate.so
LoadModule proxy_module /usr/local/apache2/modules/mod_proxy.so
LoadModule proxy_fcgi_module /usr/local/apache2/modules/mod_proxy_fcgi.so

<VirtualHost *:81>
    # Proxy .php requests to port 9000 of the php-fpm container
    # ProxyPassMatch ^/(.*\.php(/.*)?)$ fcgi://php:9000/var/www/html/$1
    DocumentRoot /var/www/html/
    <Directory /var/www/html/>
        # DirectoryIndex index.php
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    # Send apache logs to stdout and stderr
    CustomLog /proc/self/fd/1 common
    ErrorLog /proc/self/fd/2
</VirtualHost>

---

# nginx/prod/prod.conf
user  nginx;
worker_processes  1;

events {
  worker_connections  1024;
}

http {
  include /etc/nginx/mime.types;
  client_max_body_size 100m;

  upstream backend {
    server backend:81;
  }


  server {
    listen 80;
    charset utf-8;

    root /dist/;
    index index.html;

    location /backend {
      proxy_redirect off;
      proxy_pass http://backend;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
    }


  }
}

---

# nginx/prod/Dockerfile
# build stage
FROM node:10.14.2-jessie as build-stage
WORKDIR /app/
COPY frontend/package.json /app/
RUN npm cache verify
RUN npm install
COPY frontend /app/
RUN npm run build

# production stage
FROM nginx:1.13.12-alpine as production-stage
COPY nginx/prod/prod.conf /etc/nginx/nginx.conf
COPY --from=build-stage /app/dist /dist/
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Edit:

docker-compose exec backend netstat -lnpt

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.11:38317        0.0.0.0:*               LISTEN      -
tcp        0      0 :::80                   :::*                    LISTEN      1/httpd

docker-compose exec nginx sh -c "nc backend 81 && echo opened || echo closed"

closed.

Answer 1

docker-compose exec backend netstat -lnpt shows us that the httpd webserver for service backend is listening on port 80 and not 81.

So must probably, your Dockerfile apache/Dockerfile is incorrect regarding how it tries to provide your custom httpd configuration apache/apache.conf.

To investigate further:

• Make sure the main apache conf contents is what you expect with: docker-compose exec backend cat /usr/local/apache2/conf/httpd.conf
• Inspect your backend service log: docker-compose logs backend

Doing so, you will realize your are missing the Listen 81 directive in the main apache config file. You can fix this in your apache/Dockerfile file:

# apache/Dockerfile
FROM httpd:2.4.32-alpine

RUN apk update; \
    apk upgrade;

# Copy apache vhost file to proxy php requests to php-fpm container
COPY apache/apache.conf /usr/local/apache2/conf/apache.conf

RUN echo "Listen 81" >> /usr/local/apache2/conf/httpd.conf
RUN echo "Include /usr/local/apache2/conf/apache.conf" >> /usr/local/apache2/conf/httpd.conf

---

Why have your backend container listen on port 81?

It does not add any value to make your different containers open different ports. Each container has it's own IP address, thus there is no need for avoiding port collision between the services defined in a docker-compose project.