Azure Load Balancer with Haproxy causing 504 timeouts

Question

We use 2 active haproxy servers behind Azure's Load Balancer offering which then distribute load to our front end web cluster. We have this setup because we need HA on our haproxy servers and we also use a number of rules in haproxy for routing traffic which are not supported in any of the Azure offerings.

We have come across a problem where traffic hits the azure load balancer, is routed to the haproxy box and then sent onto IIS to be processed but we never receive a response back from the server but I can see the request has been processed as data has been written to the database. I have a log in the haproxy logs of a 504 but when I check IIS logs nothing is registered so it gets processed in IIS but it looks like the connection is dropped at some stage.

I tested this setup without the Azure Load Balancer and just Haproxy on its own works fine but when we introduce the Azure Load Balancer we start to get 504 errors. Has any one seen such an issue?

haproxy.cfg

global
    log 127.0.0.1 local0
    maxconn 4096
    user haproxy
    group haproxy
    tune.ssl.default-dh-param 2048
    tune.maxrewrite 4096

    ssl-default-bind-options no-sslv3
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

    ssl-default-server-options no-sslv3
    ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

defaults
    log global
    option httplog
    mode http
    option dontlognull
    option tcplog
    retries 3
    option redispatch
    maxconn 20000
    timeout connect 8000
    timeout client 50000
    timeout server 50000

frontend release-micro-http
    bind :8082
    reqadd X-Forwarded-Proto:\ http
    default_backend release-micro-backend

backend release-micro-backend
    balance roundrobin
    option http-server-close
    option forwardfor
    option httpchk GET /check.txt
    server worker1 10.1.2.1:8086 check
    server worker2 10.1.2.2:8086 check

There's a 4-character code shown in HAProxy log entries. For normal, successful requests, it is set to ----, but for all others, it should be something different. What's the code for these failed requests? — Michael - sqlbot

Ollie Ollie · Accepted Answer · 2019-02-27T09:25:55

As @sqlbot said, I noticed that the Haproxy Log was showing -sH- as the termination state

s = the server-side timeout expired while waiting for the server to send or receive data.

H = the proxy was waiting for complete, valid response HEADERS from the server (HTTP only).

We looked at the httperr.log in C:\Windows\system32\LogFiles\Httperr and noticed we were getting:

POST /api/v1/clients - - 2 Connection_Dropped

I recycled the app pool and enabled request tracing and the application started responding correctly again. We have idle timeout set to 0 so we will need to investigate our IIS configuration and then application setup further.

Azure Load Balancer with Haproxy causing 504 timeouts

1 Answers