Exposing Nifi UI when running a cluster in Docker Swarm

Question

I'm running an Apache Nifi cluster on docker swarm using below configuration:

version: '3'

services:

  zookeeper:
    hostname: zookeeper
    image: 'bitnami/zookeeper:latest'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  nifi:
    image: apache/nifi:latest
    ports:
      - 8080
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ELECTION_MAX_WAIT=1 min

This works fine and I can easily scale up the number of Nifi instances.

However, when trying to access the Nifi UI via the published port, it doesn't seem to work. I get a connection refused when trying to access it via any of the swarm nodes.

ID                  NAME            MODE                REPLICAS      IMAGE                                   PORTS
klp9kjm7jwdy        nifi            replicated          3/3           apache/nifi:latest                      *:30003->8080/tcp
qa3rf9pi6uyw        zookeeper       replicated          1/1           bitnami/zookeeper:latest

The problem seems to be related to the fact that Nifi is binding to the hostname for the host it runs on. Causing it to be only available inside the swarm network by using it's container id.

This does work from within any container inside the swarm network, but not via the published port.

I also tried configuring NIFI_WEB_HTTP_HOST=0.0.0.0 to make sure Nifi binds to all network interfaces, but that breaks communication between the instances in the cluster.

How should I configure Nifi/Docker swarm for being able to properly access Nifi's UI through the swarm routing mesh network?

wjans wjans · Accepted Answer · 2020-06-17T06:15:48

Managed to get it up and running with the help of the Apache Nifi Dev Mailing List.

The problem lies in the fact that Nifi by default does not bind to all network interfaces and traffic from the swarm network uses a different network interface. Worked by adding other network interfaces to the Nifi configuration too.

My Nifi container has three network interfaces eth0, eth1 and eth2, so I added these properties to ${NIFI_HOME}/conf/nifi.properties:

nifi.web.http.network.interface.eth0=eth0
nifi.web.http.network.interface.eth1=eth1
nifi.web.http.network.interface.eth2=eth2

I suppose this could be narrowed down to only add the network interface being used for traffic from the Swarm ingress network.

Exposing Nifi UI when running a cluster in Docker Swarm

1 Answers