Elk-stack- Filebeat issue on windows

Question

I'm having an issue with filebeat on windowsXP. There is a csv file that is written (every 10/15 seconds) by an application that collect data measurement. Filebeat reads this file every time from scratch and it causes duplicated rows. Any advices?

filebeat.prospectors:
- input_type: log
  paths:
    - C:\nms2k\ems\measure\PERF*.csv
  include_lines: ['RTT','RTJ']
  logging.level: debug
  #----------------------------- Logstash output --------------------------------
  output.logstash:
      hosts: ["xxx.xxx.xxx.xxx:1000"]

Here my registry.log: https://pastebin.com/MbPVgH5S

and here my filebeat.log: https://pastebin.com/A8tqukQT

Can you activate the debug mode in filebeat ? Add : logging.level: debug at the end of the fileabeat.yml You can clean the registry entries (if you're testing and not willing to keep them) and start your filebeat and see what happens — YellowStrawHatter
i put log in debug mode already i guess, as you can see : logging.level: debug #----------------------------- Logstash output -------------------------------- it's not at the end but should be fine, i tried to delete regestry with the same result — Parisone
Sorry did'nt see it. Alright, in your logstash output add the following : stdout { codec => rubydebug } Logstash will debug all incoming logs from Filebeat. Then clean all the ELK stack (delete the exisiting index or create a new index for testing and clean registry file) and see what it's actually coming.Of course, you need to stop filebeat for a while and then prepare your log file (try to create only one entry) — YellowStrawHatter
This step will make sure to identify the problem source, whether it's Filebeat or Logstash is the responsible of the duplication. In /var/log/logstash.stdout tell me how many entries did you Logstash receive, if it's only 1 then the problem will be in the Logstash side. — YellowStrawHatter

tsg tsg · Accepted Answer · 2017-10-25T10:46:51

From the registry file it looks like the file ID is changing every time. I suspect this happens because the application rewrites the whole file, instead of just appending to it. If that's the case, then I'm afraid that Filebeat doesn't currently support that, so check if it can be changed on the application side.

Elk-stack- Filebeat issue on windows

1 Answers