Why does Traefik not proxy new services in a Docker Swarm?

Question

I try to setup traefik with a Docker Swarm. I have to VMs - one manger-node and one worker-node. In addition I have created a external network with:

docker network create --driver=overlay proxy-net

I start traefik as a service within my manager-node with the following docker-compose.yml file:

version: '3'

services:
  traefik:
     image: traefik:v1.4.4
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
       - $PWD/management/traefik/traefik.toml:/etc/traefik/traefik.toml
     ports:
       - 80:80
       - 8100:8080
     deploy:
       placement:
         constraints:
           - node.role == manager

networks:
   default:
    external:
      name:  proxy-net

My traefik.toml file looks like this:

Debug : "DEBUG"
defaultEntryPoints = ["http"]
[entryPoints]
    [entryPoints.http]
    address = ":80"

[web]
address = ":8080"

[docker]
watch = true
swarmmode = true
domain = "mydomain.com"
exposedbydefault = true

When I now start a new service (e.g. emilevauge/whoami) with:

docker service create \
    --name whoami1 \
    --publish mode=host,target=80,published=8002 \
    --network proxy-net \
    --label traefik.docker.network=proxy-net \
    --label traefik.frontend.rule=Host:whoami.mydomain.com \
    --label traefik.port=8002 \
    emilevauge/whoami

The service is seen by the traefik web frontend. So at first every thing looks fine. I can access the service directly on my worker node on port 8002. But traefik does not seem to be able to proxy this service. When I browse my endpoint URL (whomai.mydomain.com) I get the answer:

Bad Gateway

The traefik logfile (logLevel=DEBUG) shown messages like this:

proxy_traefik.1.zl50yv6got5f@tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Filtering container without port and no traefik.port label swarmpit_app.1 : strconv.Atoi: parsing "": invalid syntax"
proxy_traefik.1.zl50yv6got5f@tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Filtering container without port and no traefik.port label proxy_traefik.1 : strconv.Atoi: parsing "": invalid syntax"
proxy_traefik.1.zl50yv6got5f@tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Filtering container without port and no traefik.port label swarmpit_db.1 : strconv.Atoi: parsing "": invalid syntax"
proxy_traefik.1.zl50yv6got5f@tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Validation of load balancer method for backend backend-whoami1-whoami1-whoami1 failed: invalid load-balancing method ''. Using default method wrr."
proxy_traefik.1.zl50yv6got5f@tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Configuration received from provider docker: {"backends":{"backend-whoami1-whoami1-whoami1":{"servers":{"service-0":{"url":"http://10.0.1.5:8002","weight":0}},"loadBalancer":{"method":"wrr"}}},"frontends":{"frontend-whoami1-whoami1-whoami1":{"entryPoints":["http"],"backend":"backend-whoami1-whoami1-whoami1","routes":{"service-whoami1":{"rule":"Host:whoami.mydomain.com"}},"passHostHeader":true,"priority":0,"basicAuth":[],"headers":{}}}}"

I played around several hours with different configurations. I also read the very concise documentation about traefik and docker-swarm. But I don't get any idea what I'm doing wrong.

Can any body help me with some tips how to better understand the problem?

Answer 1

I think it is not working because you Træfik service is not on the same docker network as your whoami1 service.

You should try to add proxy-net network to your Træfik service in your compose file.

There is a warning in Træfik documentation at the end of this page https://docs.traefik.io/configuration/backends/docker/

when running inside a container, Træfik will need network access through:
docker network connect <network> <traefik-container>

Answer 2

As already mentioned, they need to be in the same overlay network which is not ingress. The ingress network is only for manager nodes.
Further more, your traefik service is not assigned to the proxy-net network. You're creating proxy-net in your traefik config part, but don't assigned it to it

version: '3'

services:
    traefik:
        image: traefik:v1.4.4
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - $PWD/management/traefik/traefik.toml:/etc/traefik/traefik.toml
        ports:
            - 80:80
            - 8100:8080
        networks:
        - proxy-net
        deploy:
        placement:
            constraints:
                - node.role == manager

networks:
    proxy-net:
        driver: overlay

Further more, you should create a config with docker config create. Otherwise with $PWD/management/traefik/traefik.toml you need to copy the traefik.toml file to every manager node.

Append your compose file with

configs:
  traefik_conf_v1:
    file: ./traefik.toml

and your traefik part with

    configs:
      - source: traefik_conf_v1
        target: /etc/traefik/traefik.toml

Now back to your problem.
What's your service is missing is the label to the backend. Otherwise traefik doesn't know where the service is running (network assignment isn't enough!).

docker service create \
    --name whoami1 \
    --publish mode=host,target=80,published=8002 \
    --network proxy-net \
    --label traefik.backen=whoami1 \
    --label traefik.docker.network=proxy-net \
    --label traefik.frontend.rule=Host:whoami.mydomain.com \
    --label traefik.port=8002 \
    emilevauge/whoami

This should work. And when it does, stop publishing ports of your services. That makes everything complicated when you're in a hurry and need to scale. Remember, work balancing is handle by the swarm itself.

And yeah, dynamic flexible reverse proxys is still a problem nowadays :)
Remember, you got your entry points on manager nodes with traefik, but not on the worker nodes.

Answer 3

I finally I solved this issue. It was actually not a Traefik problem. The problem was, that both VMs from my provider have the same private IPv4 address. To register and join the docker-swarm it is important to provide the public IPv4 addresses with the option --advertise-addr

To register the swarm I have to run:

docker swarm init --advertise-addr [manager-ip-address]

to join the swarm by a worker-node also the public IPv4 address need to be set explicitly:

docker swarm join \
 --token SWMTKN-1-xxxxxxxxxxxxxxxxxxxx-xxxxxxxx \
 --advertise-addr [worker-ip-address]\
 [manager-ip-address]:2377

Answer 4

I would say that your setup of service labels was wrong. Traefik redirects requests to swarm service port so it should go to port 80, not to published port 8002. I think that correct service create command should be:

docker service create \
    --name whoami1 \
    --publish mode=host,target=80,published=8002 \
    --network proxy-net \
    --label traefik.docker.network=proxy-net \
    --label traefik.frontend.rule=Host:whoami.mydomain.com \
    --label traefik.port=80 \
    emilevauge/whoami

And publishing the 80 port for whoami service is not needed.