nginx upstream load balancing through private networking showing 404

Question

I have two droplets on Digital Ocean. One load balancer with nginx and one node/express webserver with nginx reverse proxy. Let's call them load-1 and web-1. load-1 handles SSL termination and forwards requests via nginx upstream module to web-1 via http over private networking provided by Digital Ocean.

When accessing web-1 on it's public IP everything works. When accessing through load-1 I receive only 404s. I have verified that the requests are actually forwarded to web-1, this is what the nginx access log for web-1 shows on every request received from load-1:

load-1.private.ip - [09/Jan/2017:13:14:00 +0000] "GET / HTTP/1.0" 404 580 "-" >"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) >Chrome/55.0.2883.87 Safari/537.36"

Why are forwarded requests not working when direct requests are working? Since web-1 is working when accessed directly there must be something wrong with how I forward requests from load-1 to web-1?

My nginx config on load-1:

    upstream web-servers {
        server web-1.private.ip;
    }

    server {
        listen 80;
        listen 443 ssl;
        server_name mydomain.com;

        ssl on;
        ssl_certificate /etc/ssl/mycert.crt;
        ssl_certificate_key /etc/ssl/mykey.key;
        ssl_session_cache shared:SSL:20m;
        ssl_session_timeout 10m;
        ssl_prefer_server_ciphers on;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        add_header Strict-Transport-Security "max-age=31536000";

        location / {
            proxy_pass http://web-servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
   }

My nginx config on web-1:

   server {
       listen 80;

       server_name web-1.public.ip web-1.private.ip;

       location / {
           proxy_pass http://127.0.0.1:5000;
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection 'upgrade';
           proxy_set_header Host $host;
           proxy_set_header X-Forwarded-For $remote_addr;
           proxy_cache_bypass $http_upgrade;
       }
   }

Kyle Kyle · Accepted Answer · 2017-01-09T16:09:52

Simply, Nginx on web-1 doesn't know what configuration to use.

Nginx looks at the host header to determine the server configuration to use. You're setting the host to be mydomain.com in the proxy settings on load-1, But there's no corresponding entry for mydomain.com on web-1.

Either

Set the default_server flag on web-1 (by changing the listen 80; directive to listen 80 default_server;)
Remove any other server blocks so this is the only block (causing Nginx to treat it as the default server)
Add mydomain.com to the server_name list

nginx upstream load balancing through private networking showing 404

1 Answers