Filebeat 7.2 - Save logs from Docker containers to Logstash

Question

I have a few Docker containers running on my ec2 instance.

I want to save logs from these containers directly to Logstash (Elastic Cloud).

When I tried to install Filebeat manually, everything worked allright. I have downloaded it using

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.2.0-linux-x86_64.tar.gz

I have unpacked it, changed filebeat.yml configuration to

filebeat.inputs:

- type: log

  enabled: true

  fields:
    application: "myapp"

  fields_under_root: true

  paths:
    - /var/lib/docker/containers/*/*.log

cloud.id: "iamnotshowingyoumycloudidthisisjustfake"
cloud.auth: "elastic:mypassword"

This worked just fine, I could find my logs after searching application: "myapp" in Kibana.

However, when I tried to run Filebeat from Docker, no success.

This is filebeat part of my docker-compose.yml

filebeat:
    image: docker.elastic.co/beats/filebeat:7.2.0
    container_name: filebeat
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock #needed for autodiscover

My previous filebeat.yml from manual execution doesn't work, so I have tried many examples, but nothing worked. This is one example which I think should work, but it doesn't. Docker container starts no problem, but it can't read from logfiles somehow.

filebeat.autodiscover:
  providers:
    - type: docker

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log
  json.keys_under_root: true
  json.add_error_key: true
  fields_under_root: true
  fields:
    application: "myapp"

cloud.id: "iamnotshowingyoumycloudidthisisjustfake"
cloud.auth: "elastic:mypassword"

I have also tried something like this

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        config:
          - type: docker
            containers.ids:
              - "*"

filebeat.inputs:
  - type: docker
    containers.ids:
      - "*"
    processors:
      - add_docker_metadata:
    fields:
      application: "myapp"
    fields_under_root: true

cloud.id: "iamnotshowingyoumycloudidthisisjustfake"
cloud.auth: "elastic:mypassword"

I have no clue what else to try, filebeat logs still shows

 "harvester":{"open_files":0,"running":0}}

I am 100% sure that logs from containers are under /var/lib/docker/containers/*/*.log ... as I said, Filebeat worked, when installed manually, not as docker image.

Any suggesions ?

Output log from Filebeat

2019-07-23T05:35:58.128Z        INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.2.0
2019-07-23T05:35:58.128Z        INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'filebeat-7.2.0' as ILM is enabled.
2019-07-23T05:35:58.129Z        INFO    elasticsearch/client.go:166     Elasticsearch url: https://123456789.us-east-1.aws.found.io:443
2019-07-23T05:35:58.129Z        INFO    [publisher]     pipeline/module.go:97   Beat name: e3e5163f622d
2019-07-23T05:35:58.136Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-07-23T05:35:58.142Z        INFO    instance/beat.go:421    filebeat start running.
2019-07-23T05:35:58.142Z        INFO    registrar/migrate.go:104        No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2019-07-23T05:35:58.142Z        INFO    registrar/migrate.go:112        Initialize registry meta file
2019-07-23T05:35:58.144Z        INFO    registrar/registrar.go:108      No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2019-07-23T05:35:58.146Z        INFO    registrar/registrar.go:145      Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2019-07-23T05:35:58.146Z        INFO    registrar/registrar.go:152      States Loaded from registrar: 0
2019-07-23T05:35:58.146Z        INFO    crawler/crawler.go:72   Loading Inputs: 1
2019-07-23T05:35:58.146Z        WARN    [cfgwarn]       docker/input.go:49      DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2019-07-23T05:35:58.150Z        INFO    log/input.go:148        Configured paths: [/var/lib/docker/containers/*/*.log]
2019-07-23T05:35:58.150Z        INFO    input/input.go:114      Starting input of type: docker; ID: 11882227825887812171
2019-07-23T05:35:58.150Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2019-07-23T05:35:58.150Z        WARN    [cfgwarn]       docker/docker.go:57     BETA: The docker autodiscover is beta
2019-07-23T05:35:58.153Z        INFO    [autodiscover]  autodiscover/autodiscover.go:105        Starting autodiscover manager
2019-07-23T05:36:28.144Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        
{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":17}},"total":{"ticks":40,"time":{"ms":52},"value":40},"user":{"ticks":30,"time":{"ms":35}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":9},"info":{"ephemeral_id":"4427db93-2943-4a8d-8c55-6a2e64f19555","uptime":{"ms":30111}},"memstats":{"gc_next":4194304,"memory_alloc":2118672,"memory_total":6463872,"rss":28352512},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":1},"load":{"1":0.31,"15":0.03,"5":0.09,"norm":{"1":0.31,"15":0.03,"5":0.09}}}}}}

Do you have the - "./var/lib/docker/containers:/var/lib/docker/containers:ro" volume mount in your compose file? My initial guess is that you are not mounting the containers log directories into your filebeat Docker container so it isn't finding the log files. — markybb41
@markybb41 what I thought as well however “/var/...” not “./var/...” (unless you’re running docker-compose from the root directory ofc). — masseyb
I am running docker-compose in home directory, but "./var/..." didn't solve the problem for me. — Tomas Lukac
I have updated my docker-compose, I was mounting the /var/lib/docker/containers but didn't write it here. — Tomas Lukac

justkind justkind · Accepted Answer · 2019-07-24T18:16:02

Hmm, I don't see anything obvious in the Filebeat config on why its not working, I have a very similar config running for a 6.x Filebeat.

I would suggest doing a docker inspect on the container and confirming that the mounts are there, maybe check on permissions but errors would have probably shown in the logs.

Also could you try looking into using container input? I believe it is the recommended method for container logs in 7.2+: https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-input-container.html

Filebeat 7.2 - Save logs from Docker containers to Logstash

1 Answers