pfSense + HAProxy – Reverse Proxy with multiple Services on one internal IP

3
votes

currently I am using pfSense on my server with the HAProxy package, because I can easily configure it via the GUI.

I configured HAProxy to act as a reverse proxy corresponding to this guide: https://blog.devita.co/pfsense-to-proxy-traffic-for-websites-using-pfsense/

SSL offloading works like a charm. The problem I have is when I have more than one service (open port) on the same internal IP it seems not to be working.

Example:

  • I configure service1.domain.com for Service1 with port 8000 (10.100.10.101:8000) and it works flawlessly.
  • Now I need another port on the same machine (e.g. 10.100.10.101:8082) with another service. If I configure another backend pointing to the same IP but with a different port I can only reach the second servce (service2.domain.com) even if I access service1.domain.com.

My use case is that I am trying to set up Seafile which is using port 8000 for the web GUI and port 8082 for the fileserver. Right now I am able to access the web GUI but I am not able to upload, download or share files.

My configuration:


    # Automaticaly generated, dont edit manually.
    # Generated on: 2018-09-29 19:24
    global
        maxconn    1000
        stats socket /tmp/haproxy.socket level admin 
        gid    80
        nbproc    1
        hard-stop-after    15m
        chroot    /tmp/haproxy_chroot
        daemon
        tune.ssl.default-dh-param    8192
        server-state-file /tmp/haproxy_server_state
        ssl-default-bind-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
        ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
        ssl-default-server-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
        ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

    listen HAProxyLocalStats
        bind 127.0.0.1:2200 name localstats
        mode http
        stats enable
        stats admin if TRUE
        stats show-legends
        stats uri /haproxy/haproxy_stats.php?haproxystats=1
        timeout client 5000
        timeout connect 5000
        timeout server 5000

    frontend shared-frontend-merged
        bind    X.X.X.X:443 name X.X.X.X:443   ssl crt-list /var/etc/haproxy/shared-frontend.crt_list  
        mode    http
        log    global
        option    http-keep-alive
        option    forwardfor
        acl https ssl_fc
        http-request set-header    X-Forwarded-Proto http if !https
        http-request set-header    X-Forwarded-Proto https if https
        timeout client    30000
        http-response set-header Strict-Transport-Security max-age=15768000
        acl    aclcrt_shared-frontend    var(txn.txnhost) -m reg -i ^([^\.]*)\.domain\.com(:([0-9]){1,5})?$
        acl    ACL1    var(txn.txnhost) -m str -i test.domain.com
        acl    ACL2    var(txn.txnhost) -m str -i service1.domain.com
        acl    ACL3    var(txn.txnhost) -m str -i service2.domain.com
        http-request set-var(txn.txnhost) hdr(host)
        default_backend test.domain.com_ipv4
        default_backend service1.domain.com_ipvANY
        default_backend service2.domain.com_ipvANY

    frontend http-to-https
        bind    X.X.X.X:80 name X.X.X.X:80   
        mode    http
        log    global
        option    http-keep-alive
        timeout client    30000
        http-request redirect scheme https 

    backend test.domain.com_ipv4
        mode    http
        id    10100
        log    global
        timeout connect    30000
        timeout server    30000
        retries    3
        source ipv4@ usesrc clientip
        option    httpchk GET / 
        server    testvm-server01 10.100.10.101:54080 id 10101 check inter 1000  

    backend service1.domain.com_ipvANY
        mode    http
        id    102
        log    global
        timeout connect    30000
        timeout server    30000
        retries    3
        option    httpchk GET / 
        server    seafile-vm-01 10.100.10.103:8000 id 101 check inter 1000  

    backend service2.domain.com_ipvANY
        mode    http
        id    104
        log    global
        timeout connect    30000
        timeout server    30000
        retries    3
        option    httpchk GET / 
        server    seafile-vm-02 10.100.10.103:8082 id 103 check inter 1000

I would really be glad if anyone can point me in the right direction, thank you in advance and if you need further information please tell me.

Best regards,

Bioneye

1
linuxreverse-proxyhaproxypfsenseseafile-server
This question seems to be more relevant for Server Fault. Perhaps a moderator can move it? – MechMK1

1 Answers

1
votes

I was able to solve my problem with the help of one awesome user over on reddit.

The first problem was that I misconfigured my frontend and thus had 3 default_backends. That was the reason why every services pointed to the same virtual machine. To solve it I just had to add the if condition corresponding to my ACL name.

The second problem was that my Service2 was shown as DOWN on the HAProxy stats page. I had to change the health check method from HTTP to Basic and that finally resolved everything.

This is the working configuration:

# Automaticaly generated, dont edit manually.
# Generated on: 2018-10-02 16:59
global
    maxconn         1000
    stats socket /tmp/haproxy.socket level admin 
    gid         80
    nbproc          1
    hard-stop-after     15m
    chroot              /tmp/haproxy_chroot
    daemon
    tune.ssl.default-dh-param   8192
    server-state-file /tmp/haproxy_server_state
    ssl-default-bind-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-server-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

listen HAProxyLocalStats
    bind 127.0.0.1:2200 name localstats
    mode http
    stats enable
    stats admin if TRUE
    stats show-legends
    stats uri /haproxy/haproxy_stats.php?haproxystats=1
    timeout client 5000
    timeout connect 5000
    timeout server 5000

frontend shared-frontend-merged
    bind            X.X.X.X:443 name X.X.X.X:443   ssl crt-list /var/etc/haproxy/shared-frontend.crt_list  
    mode            http
    log         global
    option          http-keep-alive
    option          forwardfor
    acl https ssl_fc
    http-request set-header     X-Forwarded-Proto http if !https
    http-request set-header     X-Forwarded-Proto https if https
    timeout client      30000
    http-response set-header Strict-Transport-Security max-age=15768000
    acl         aclcrt_shared-frontend  var(txn.txnhost) -m reg -i ^([^\.]*)\.domain\.com(:([0-9]){1,5})?$
    acl         ACL1    var(txn.txnhost) -m beg -i test.domain.com
    acl         ACL2    var(txn.txnhost) -m beg -i service1.domain.com
    acl         ACL3    var(txn.txnhost) -m beg -i service2.domain.com
    http-request set-var(txn.txnhost) hdr(host)
    use_backend test.domain.com_ipv4  if  ACL1 
    use_backend service1.domain.com_ipvANY  if  ACL2 
    use_backend service2.domain.com-seafhttp_ipvANY  if  ACL3 

frontend http-to-https
    bind            X.X.X.X:80 name X.X.X.X:80   
    mode            http
    log         global
    option          http-keep-alive
    timeout client      30000
    http-request redirect scheme https 

backend test.domain.com_ipv4
    mode            http
    id          10100
    log         global
    timeout connect     30000
    timeout server      30000
    retries         3
    source ipv4@ usesrc clientip
    option          httpchk GET / 
    server          testvm-server01 10.100.10.101:54080 id 10101 check inter 1000  

backend service1.domain.com_ipvANY
    mode            http
    id          102
    log         global
    timeout connect     30000
    timeout server      30000
    retries         3
    option          httpchk GET / 
    server          seafile-vm-01 10.100.10.103:8000 id 101 check inter 1000  

backend service2.domain.com-seafhttp_ipvANY
    mode            http
    id          104
    log         global
    timeout connect     30000
    timeout server      30000
    retries         3
    server          seafile-vm-02 10.100.10.103:8082 id 103 check inter 1000

For further details: https://www.reddit.com/r/PFSENSE/comments/9kezl3/pfsense_haproxy_reverse_proxy_with_multiple/?st=jmruoa9r&sh=26d24791

TLDR: I misconfigured my Action Table and had the wrong health check in place.

Greetings,

Bioneye