Docker Logging Efk 7.10.1 Compose ECONNREFUSED

Question

just trying to build a test app for learning purposes on how to collect Docker logs to EFK (Elasticsearch7.10.1 + Fluentd + Kibana7.10.1) stack.

Elastic starts up fine and is reachable from http://localhost:5601/

But fluentd-* is not available as an index pattern, I assume do to the error I am getting on the logs from kibana:

kibana_1         | {"type":"log","@timestamp":"2021-01-03T23:46:32Z","tags":["error","elasticsearch","monitoring"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_xpack => connect ECONNREFUSED 172.20.0.3:9200"}
kibana_1         | {"type":"log","@timestamp":"2021-01-03T23:46:32Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2021-01-03T23:46:32Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2021-01-03T23:46:32Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
kibana_1         | {"type":"log","@timestamp":"2021-01-03T23:46:32Z","tags":["warning","plugins","monitoring","monitoring"],"pid":6,"message":"X-Pack Monitoring Cluster Alerts will not be available: No Living connections"}
kibana_1         | {"type":"log","@timestamp":"2021-01-03T23:46:32Z","tags":["error","elasticsearch","data"],"pid":6,"message":"[ConnectionError]: connect ECONNREFUSED 172.20.0.3:9200"}
kibana_1         | {"type":"log","@timestamp":"2021-01-03T23:46:32Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}

172.20.0.3:9200 and http://elasticsearch:9200/ are not reachable through browser

http://localhost:9200/ is reachable

What am I missing? I have been working on this for a week and don't know where to look anymore, thanks!

Docker-compose.yml

version: '2'
services:
  web:
    image: httpd
    ports:
      - "8080:80"
    links:
      - fluentd
    logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
        tag: httpd.access

  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf
    links:
      - "elasticsearch"
    ports:
      - "24224:24224"
      - "24224:24224/udp"

  elasticsearch:
    image: elasticsearch:7.10.1
    environment: 
      - "network.host=0.0.0.0"
      - "transport.host=127.0.0.1"
    expose:
      - 9200
    ports:
      - "9200:9200"

  kibana:
    image: kibana:7.10.1
    environment:
      server.host: 0.0.0.0
      elasticsearch.hosts: http://localhost:9200
    ports:
      - "5601:5601"

Dockerfile

# fluentd/Dockerfile
FROM fluent/fluentd:v1.11.5-debian-1.0

RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "4.0.4"]

fluentd.conf file

# fluentd/conf/fluent.conf
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match *.**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>
  <store>
    @type stdout
  </store>
</match>

ItayB ItayB · Accepted Answer · 2021-01-04T06:46:11

This is totally fine and the expected outcome.

In docker, if you want your service (Kibana) to be available from the localhost you should map it's port to localhost. You are doing that by:

    ports:
      - "5601:5601"

then you can access Kibana from your browser (localhost) by using http://localhost:5601

On other hand, internally, if you want to access one container from another you should use the container name (rather than localhost) - so if you want to access the Kibana within the elasticsearch container you would execute into the elasticsearch container and call:

curl http://kibana:5601

EDIT:

one interesting case is your web container that uses a different port internally and externally, so from localhost you would:

curl http://localhost:8080

while internally (within that docker network) you will access by:

http://web

(you can omit the 80 since its the default http port)

EDIT2:

As stated in the documentation, the default value for elasticsearch.hosts is http://elasticsearch:9200.

Docker Logging Efk 7.10.1 Compose ECONNREFUSED

1 Answers