Haproxy multiple instances - service backends don't stabilize for 5 min

Question

Issue:
Using SRV records in Haproxy the backends go DOWN and UP a few times but don't stabilize for about 5 min

Architecture:
Cloud Provider: AWS
Running multiple (3) Haproxy containers on ECS with an ELB to distribute the traffic between them.
I setup Haproxy backend to work with SRV records for service discovery
I dynamically want to add services (backends) to Haparoxy so I use a config template and the socket reload.
Each service has 2 backends

The reload script looks like this, it generates the template and reloads if the template changes: #no new services were added so I don't think the reload script has anything to do with the issue

#!/bin/bash
mkdir -p /var/PROJECT/services/
aws s3 cp s3://$BUCKET/services/services.json /var/PROJECT/services/
python3 /var/PROJECT/docker/generateHaproxyConfig.py >> /var/log/haproxy-gen.log
if ! diff -q /etc/haproxy/haproxy.cfg.new /etc/haproxy/haproxy.cfg &>/dev/null; then
    mv /etc/haproxy/haproxy.cfg.new /etc/haproxy/haproxy.cfg
    mkdir -p /etc/haproxy/states
    for b in $(socat /var/run/haproxy/socket - <<< "show backend" | fgrep -v '#')
    do
        socat /var/run/haproxy/socket - <<< "show servers state $b" > /etc/haproxy/states/$b
    done
    /etc/init.d/haproxy reload
fi

In my test I have the service running (06:49:14) and then stop it (~06:50:00) then start it a few seconds after.

In the logs HA1 we see it did the following:
1) was UP at 06:49:14
2) was DOWN at 06:50:16
3) backend1 was UP at 06:51:17 #it should have had both backends UP here and no more DOWNs
4) backend1 went DOWN at 06:51:58 #this is unexpected!
5) backend1 is back UP at 06:52:09
6) backend2 is up at 06:56:10 #why did this take so long?

The other containers had similar behaviour

The SRV DNS records look like this:
_test-git._tcp.staging.qa. SRV 1 1 32773 6c733da247894543afe5d11828751a05._test-git._tcp.staging.qa.
_test-git._tcp.staging.qa. SRV 1 1 32773 ec40447095d345069daf9ff7c0b6bd8c._test-git._tcp.staging.qa.
6c733da247894543afe5d11828751a05._test-git._tcp.staging.qa. A 172.18.2.194
ec40447095d345069daf9ff7c0b6bd8c._test-git._tcp.staging.qa. A 172.18.1.171

Haproxy config template:

global
    log /dev/log    local0 info
    #log /dev/log   local1 info
    #log /dev/log   local2 info
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    pidfile /run/haproxy.pid

    stats socket /var/run/haproxy/socket
    server-state-base /etc/haproxy/states
    # stats socket /var/run/hapee-lb.sock mode 666 level admin
    # stats socket [email protected]:9999 level admin
    stats timeout 30s
    daemon

    spread-checks 4
    tune.maxrewrite 1024
    # tune.ssl.default-dh-param 2048

resolvers awsdns
    nameserver default {{settings.NAMESERVER}}
    resolve_retries       3
    timeout resolve       3s
    timeout retry         3s
    hold other           30s
    hold refused         30s
    hold nx              1s
    hold timeout         30s
    hold valid           10s
    hold obsolete        30s

defaults
    log     global
    mode http
    balance roundrobin

    load-server-state-from-file local

    option httplog
    option dontlognull
    # option forwardfor except 127.0.0.0/8

    option  dontlog-normal
    option socket-stats
    # option forceclose
    option forwardfor if-none
    # option httpclose
    # option forceclose
    # option http-server-close
    option  redispatch

    retries 3

    maxconn 500000
    timeout connect 5s
    timeout client  300s
    timeout server  1h
    timeout queue   30s
    # timeout http-request 5s
    # timeout http-keep-alive 15s

frontend www-http
    bind 0.0.0.0:80
    # reqadd X-Forwarded-Proto:\ http
    acl is_valid_base_domain hdr_end(host) -i PROJECT.io

    # acl servers_alive nbsrv(www-backend-%[req.hdr(host),lower,field(1,'.')]) ge 1
    # use_backend www-backend-%[req.hdr(host),lower,field(1,'.')]-redirect if is_valid_base_domain !servers_alive
    use_backend www-backend-%[req.hdr(host),lower,field(1,'.')] if is_valid_base_domain
    use_backend www-health if !is_valid_base_domain

    default_backend www-backend

frontend stats
    bind *:1936
    mode http
    option httpclose
    stats enable
    stats uri /
    stats show-legends
    #stats show-desc
    stats show-node
    stats refresh 2

{% for service in services %}

backend www-backend-{{service.name}}
    option httpchk GET /health.php
    http-check expect string 1
    acl servers_alive nbsrv(www-backend-{{service.name}}) ge 1
    http-request redirect code 302 location https://{{settings.REDIRECT_HOST}}/site/{{service.id}}/launch?s_r_path=%[path]&s_r=1&%[query] if ! servers_alive
    server-template {{service.name}} 2 _{{service.name}}._tcp.{{settings.SERVICE_DISCOVERY_DOMAIN_NAME}} resolvers awsdns resolve-prefer ipv4 resolve-opts allow-dup-ip check
{% endfor %}


backend www-backend
    # errorfile 502 /var/PROJECT/lb/statuses/200.http
    # errorfile 503 /var/PROJECT/lb/statuses/200.http

backend www-health
    acl is_valid_base_domain hdr_end(host) -i PROJECT.io
    errorfile 502 /var/PROJECT/lb/statuses/200.http
    errorfile 503 /var/PROJECT/lb/statuses/200.http

Haproxy container logs: https://gist.github.com/amitaymolko/5865e9fdef6bf47ffc2323c813edd40d

EDIT: Updated haproxy config

David Lowes David Lowes · Accepted Answer · 2019-06-30T18:22:43

My initial guess would be that it is a DNS issue, maybe you cached an "NXDOMAIN"? Try reducing the TTL for staging.qa

Haproxy multiple instances - service backends don't stabilize for 5 min

1 Answers