filebeat ignore logiles in multible prospectors

Question

I try to configure a filebeat with multible prospectors. Filebeat register all of the prospectors but ignores the localhost log files from appA and the log files from appB

My filebeat.yml:

    filebeat.prospectors:
    - type: log
      paths:
        - /vol1/appA_instance01/logs/wrapper_*.log
        - /vol1/appA_instance02/logs/wrapper_*.log
      fields: 
         log_type: "appAlogs"
         environment: "stage1"
      exclude_files: [".gz$"]
    - type: log
      paths:
        - /vol1/appA_instance01/logs/localhost.*.log
        - /vol1/appA_instance02/logs/localhost.*.log
      fields: 
         log_type: "localhostlogs"
         environment: "stage1"
      exclude_files: [".gz$"]
    - type: log
      paths:
        - /vol1/appB_instance01/logs/*.log
        - /vol1/appB_instance02/logs/*.log
      fields: 
         log_type: "appBlogs"
      environment: "stage1"
      exclude_files: [".gz$"]
   output.logstash:
   hosts: ["<HOST>:5044"]

The filebeat log file:

2017-11-15T17:32:56+01:00 INFO Home path: [/usr/share/filebeat] Config   path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017-11-15T17:32:56+01:00 INFO Setup Beat: filebeat; Version: 5.6.3
2017-11-15T17:32:56+01:00 INFO Max Retries set to: 3
2017-11-15T17:32:56+01:00 INFO Activated logstash as output plugin.
2017-11-15T17:32:56+01:00 INFO Publisher name: host
2017-11-15T17:32:56+01:00 INFO Flush Interval set to: 1s
2017-11-15T17:32:56+01:00 INFO Max Bulk Size set to: 2048
2017-11-15T17:32:56+01:00 INFO filebeat start running.
2017-11-15T17:32:56+01:00 INFO Registry file set to: /var/lib/filebeat/registry
2017-11-15T17:32:56+01:00 INFO Loading registrar data from /var/lib  /filebeat/registry
2017-11-15T17:32:56+01:00 INFO States Loaded from registrar: 222
2017-11-15T17:32:56+01:00 INFO Loading Prospectors: 3
2017-11-15T17:32:56+01:00 INFO Starting Registrar
2017-11-15T17:32:56+01:00 INFO Start sending events to output
2017-11-15T17:32:56+01:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-11-15T17:32:56+01:00 INFO Prospector with previous states loaded: 40
2017-11-15T17:32:56+01:00 INFO Starting prospector of type: log; id: 12115431240338587115 
2017-11-15T17:32:56+01:00 INFO Harvester started for file: /vol1/appA_instance01/logs/wrapper_20171115.log
2017-11-15T17:32:56+01:00 INFO Prospector with previous states loaded: 182
2017-11-15T17:32:56+01:00 INFO Starting prospector of type: log; id: 18163435272915459714 
2017-11-15T17:32:56+01:00 INFO Prospector with previous states loaded: 0
2017-11-15T17:32:56+01:00 INFO Starting prospector of type: log; id: 16959079668827945694 
2017-11-15T17:32:56+01:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 3
2017-11-15T17:33:06+01:00 INFO Harvester started for file: /vol1/appA_instance02/logs/wrapper_20171115.log

What's the reason why filebeat ignores the logiles?

/vol1/appA_instance01/logs/localhost.*.log
/vol1/appA_instance02/logs/localhost.*.log
/vol1/appB_instance01/logs/*.log
/vol1/appB_instance02/logs/*.log

greetings niesel

Your configuration is proper, please check i think it will works. — Sagar Vaghela

kvch kvch · Accepted Answer · 2017-11-24T13:59:19

The attached log shows that all three prospectors has been started and the registry file seem to have states. Are you sure that ignored log files haven't been read before by Filebeat? Does it read new lines from those log files?

Logfiles are not reread by Filebeat. So it is possible that those files were previously read.

filebeat ignore logiles in multible prospectors

1 Answers