ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://logstash:5044)) - ELK Filebeat .NET Core 3.1 Docker

Question

I'm having a strange problem I can't work out as my problem, when searching for this error, is different. People seem to have experienced this when trying to connect Filebeat to Logstash.

However, I am trying to write logs directly to Elasticsearch but I am getting Logstash related errors even though I am not even spinning up a container in Docker Compose??

Main Docker Compose File:

version: '2.2'
services:
  filebeat:
    container_name: filebeat
    build:
      context: .
      dockerfile: filebeat.Dockerfile
    volumes:
      - ./logs:/var/log
    networks:
      - esnet
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - discovery.type=single-node
      - cluster.name=docker-
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  elastichq:
    container_name: elastichq
    image: elastichq/elasticsearch-hq
    ports:
      - 8080:5000
    environment:
      - HQ_DEFAULT_URL=http://elasticsearch:9200
      - HQ_ENABLE_SSL=False
      - HQ_DEBUG=FALSE
    networks:
      - esnet  
networks:
  esnet:

DockerFile for Filebeat

FROM docker.elastic.co/beats/filebeat:7.5.2
COPY filebeat/filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
RUN chmod 644 /usr/share/filebeat/filebeat.yml
USER filebeat

I am trying to read json logs that are already in Elasticsearch format, so after reading the docs I decided to try and write directly to Elasticsearch which seems to be valid depending on the application.

My Sample.json file:

{"@timestamp":"2020-02-10T09:35:20.7793960+00:00","level":"Information","messageTemplate":"The value of i is {LoopCountValue}","message":"The value of i is 0","fields":{"LoopCountValue":0,"SourceContext":"WebAppLogger.Startup","Environment":"Development","ApplicationName":"ELK Logging Demo"}}

My Filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.json
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: log  

#----------------------------- Elasticsearch output --------------------------------

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "sample-%{+YYYY.MM.dd}"

As stated in the title of this post, I get this message in the console:

filebeat | 2020-02-10T09:38:24.438Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://logstash:5044)): lookup logstash on 127.0.0.11:53: no such host

Then when I eventually try and visualize the data in ElasticHq, inevitably, nothing is there.

So far, I've tried using commands like docker prune just in case theres something funny going on with Docker.

Is there something I'm missing?

I've just updated my post, yes is the answer, as its getting copied into the Filebeat container via a custom Dockerfile. — user1574598

Lupanoide Lupanoide · Accepted Answer · 2020-02-10T10:12:59

You have misconfigured your filebeat.yml file. Look at this error:

Failed to connect to backoff(async(tcp://logstash:5044))

Filebeat tries to connect to logstash, beacause this is the default configuration. In fact on one hand you show a filebeat.yml file and on the other hand, you haven't mounted it on /usr/share/filebeat/filebeat.yml - look at your volumes settings

  filebeat:
    container_name: filebeat
    build:
      context: .
      dockerfile: filebeat.Dockerfile
    volumes:
      - ./logs:/var/log
    networks:
      - esnet

You should mount it. If you try to copy it inside a docker container with dockerfile - why?????is necessary reinvent the wheel and add complexity? - you should use the root user:

USER root

and add root user to your service in docker-compose.yml:

user: root

ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://logstash:5044)) - ELK Filebeat .NET Core 3.1 Docker

1 Answers