Why can't I reach my traefik dashboard via HTTPS?

Question

I am trying to run a traefik container on my docker swarm cluster. Because we are using TLS encrypted communication, I want the traefik dashboard to be available via https.

In my browser, I try to access traefik via the docker swarm manager hostname via https://my.docker.manager and therefor I mounted my hosts certificate and key into the traefik service.

When I try to open https://my.docker.manager in my browser, I get a timeout.

When I try to curl https://my.docker.manager directly on the host (my.docker.manager) I get HTTP code 403 as response

My traefik config:

debug=true
logLevel = "DEBUG"

defaultEntryPoints = ["http","https"]
[entryPoints]
 [entryPoints.http]
  address = ":80"
  [entryPoints.http.redirect]
   entryPoint = "https"

 [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]
   [[entryPoints.https.tls.certificates]]
    certFile = "/etc/traefik/certs/my.docker.manager.crt"
    keyFile = "/etc/traefik/certs/my.docker.manager.key"
   [entryPoints.https.tls.defaultCertificate]
    certFile = "/etc/traefik/certs/my.docker.manager.crt"
    keyFile = "/etc/traefik/certs/my.docker.manager.key"

[api]
 address = ":8080"

[docker]
 watch = true
 swarmMode = true

My traefik compose file:

version: "3.7"

services:
  traefik:
    image: traefik
    ports:
      - 80:80
      - 443:443
    networks:
      - devops-net
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /mnt/docker-data/secrets/certs/:/etc/traefik/certs/
    configs:
      - source: traefik.conf
        target: /etc/traefik/traefik.toml
    deploy:
      placement:
        constraints:
          - node.role == manager
      labels:
        - "traefik.docker.network=devops-net"
        - "traefik.frontend.rule=Host:my.docker.manager"
        - "traefik.port=8080"

networks:
  devops-net:
    driver: overlay
    external: true

configs:
  traefik.conf:
    external: true

As described in this article (https://www.digitalocean.com/community/tutorials/how-to-use-traefik-as-a-reverse-proxy-for-docker-containers-on-ubuntu-16-04), I expected to see the traefik dashboard, when I call https://my.docker.manager in my browser. But I only get a timeout. When using curl https://my.docker.manager I get HTTP code 403. I followed the mentioned article except two differences: 1) I did not configure credentials 2) I used my hosts own certificates instead of letsencrypt

tigu tigu · Accepted Answer · 2019-07-08T15:06:14

In the meantime I found the reason for my problem (not sure what is the best solution). For the case someone is interested, I will try to explain it.

I have a Swarm of three nodes, in the network prod.company.de

My client is in another network intranet.comnpany.de

My swarm manager is adressed by docker-manager.prod.company.de. On this host, I have deployed the traefik service, that I want to access via https://docker-manager.prod.company.de (This is port 443 and because of traefik config forwarded to the traefik dashboard on 8080 inside the container).

If I track my network traffic, I can see that the https request (https://docker-manager.prod.company.de) from my client browser reaches the server and than the traffic is forwarded to the docker_gwbridge address 17.18.0.2. But the answer does not find the way back to my client because of the docker NAT configuration.

iptables -t nat -L -v

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    8   504 MASQUERADE  all  --  any    docker_gwbridge  anywhere             anywhere             ADDRTYPE match src-type LOCAL
    0     0 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere
    0     0 MASQUERADE  all  --  any    !docker_gwbridge  172.18.0.0/16        anywhere

MASQERADE says that the source IP of the request should be replaced with the bridge IP (in my case 172.18.0.1) so that answers are routed back to this IP. IN the case above, the rule 8 504 MASQUERADE all -- any docker_gwbridge anywhere anywhere would do this, but it is limited to requests from LOCAL by ADDRTYPE match src-type LOCAL, which means that using a browser on the docker host would work, but my connections from client do not work, because the answer will not find the way back to my client address.

Currently, I added one more NAT rule to my iptables: iptables -t nat -A POSTROUTING -o docker_gwbridge -j MASQUERADE

which results in

1    52 MASQUERADE  all  --  any    docker_gwbridge  anywhere             anywhere

After that, I can see the traefik Dashboard, when opening https://docker-manager.prod.company.de in my browser on the client.

But I do not understand, why I have to do this, because I found nothing about that in any documentation and I don't think that my usecase is really rare. Thats why I would be happy, if someone could have a look at this post and maybe check if I did some other thing wrong or explain to my, why I have to do such things to get a standard usecase working.

Kind regards

Why can't I reach my traefik dashboard via HTTPS?

1 Answers