Fluent-bit unable to ship logs to fluentd in docker due to EADDRNOTAVAIL

Question

I'm trying to setup the EFK stack with fluent-bit on docker containers. While I could push the logs from fluent-bit to elasticsearch, when I tried to integrate fluentd, I'm facing issues with it. This's the exact error msg:

unexpected error error_class=Errno::EADDRNOTAVAIL error="Address not available - bind(2) for \"fluent-bit\" port 24224"

The services in my docker-compose file

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
    ports:
      - '9200:9200'
      - '9300:9300'
    volumes:
      - type: bind
        source: ./config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data
    networks:
      - efk_1
  fluentd:
    image: fluent/fluentd:${FLBV}
    ports:
      - '24224:24224'
    volumes:
      - type: bind
        source: ./config/fluent.conf
        target: /fluentd/etc/fluent.conf
        read_only: true
    networks:
      - efk_1
    depends_on:
      - elasticsearch
  fluent-bit:
    image: fluent/fluent-bit:${FBITV}
    ports:
      - '2020:2020'
    volumes:
      - type: bind
        source: ./config/fluent-bit.conf
        target: /fluent-bit/etc/fluent-bit.conf
        read_only: true
      - type: bind
        source: ./sample_logs
        target: /var/log
    networks:
      - efk_1
    depends_on:
      - fluentd

Previously I directly pushed the logs from fluent-bit to elasticsearch like this without fluentd config anywhere:

[SERVICE]
    Flush   2
    Log_Level   debug

[INPUT]
    Name    tail
    Path    /var/log/log.txt

[OUTPUT]
    Name    es
    Match   *
    Host    elasticsearch
    Port    9200

This pushed the logs to elasticsearch successfully, but now I added fluentd in between, so fluent-bit will send the logs to fluentd, which will then push to elasticsearch.

fluent-bit conf:

[SERVICE]
    Flush   2
    Log_Level   debug

[INPUT]
    Name    tail
    Path    /var/log/log.txt

[OUTPUT]
    Name    forward
    Match   *
    Host    fluentd

fluentd conf:

<source>
    @type forward
    bind fluent-bit
</source>

<match **>
    @type stdout
</match>

This's giving me errors as they're not able to detect the address even though they're part of the same docker network.

These're the errors I'm getting:

fluent-bit_1 | [2019/11/06 10:31:02] [error] [io] TCP connection failed: fluentd:24224 (Connection refused)

and

fluentd_1 | 2019-11-06 10:31:02 +0000 [error]: #0 unexpected error error_class=Errno::EADDRNOTAVAIL error="Address not available - bind(2) for \"fluent-bit\" port 24224"

Can someone please help me know where I'm making a mistake?

Answer 1

I created the next configuration: docker-compose.yaml

version: "3.7"

services:
  fluentd:
    image: fluent/fluentd:v1.7.4-1.0
    ports:
      - '24224:24224'
    volumes:
      - type: bind
        source: ./config/fluent.conf
        target: /fluentd/etc/fluent.conf
        read_only: true
  fluent-bit:
    image: fluent/fluent-bit:0.14
    ports:
      - '2020:2020'
    volumes:
      - type: bind
        source: ./config/fluent-bit.conf
        target: /fluent-bit/etc/fluent-bit.conf
        read_only: true
      - type: bind
        source: /var/log/
        target: /var/log/
    depends_on:
      - fluentd

fluent.conf

<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

<match test>
  @type stdout
</match>

fluent-bit.conf

[SERVICE]
    Flush   2
    Log_Level   debug

[INPUT]
    Name    tail
    Path    /var/log/syslog
    Tag     test

[OUTPUT]
    Name    forward
    Match   *
    Host    fluentd

In these configs fluentd run and fluent-bit able to send syslog

Answer 2

I think your fluentd config should be like:

<source>
  type forward
  bind 0.0.0.0
  port 24224
</source>

<match fluent_bit>
  type stdout
</match>

As in docs

Probably fluentd should have clear IP and not hostname in bind field.

See the issue and the error description.

Answer 3

Your fluentd config needs to bind to 0.0.0.0 on input, and ship output to ES:

<source>
    @type forward
    port 24224
    bind 0.0.0.0
</source>

<match **>
    @type copy
    <store>
      @type               elasticsearch
      host                ${ELASTICSEARCH_URL}
      port                9200
    </store>
</match>

Maybe even change your Fluent Bit output also:

[OUTPUT]
    Name    forward
    Match   *
    Host    0.0.0.0
    Port    24224

If you can get that to work, then maybe tweak the settings to call the container by name and port