Filebeat duplicating events

2
votes

I am running a basic elk stack setup using Filebeat > logstash > elasticsearch > kibana - all on version 5.2

When I remove Filebeat and configure logstash to look directly at a file, it ingests the correct number of events.

If I delete the data and re-ingest the file using Filebeat to pass the same log file contents to logstash, I get over 10% more events created. I have checked a number of these to confirm the duplicates are being created by filebeat.

Has anyone seen this issue? or have any suggestions why this would happen?

1
elasticsearchlogstashelastic-stackfilebeat
Just to be clear, is it "10% more events" than logstash created, or "10% more events" than exist in the source file? – Alain Collins

1 Answers

2
votes

I need to understand first what do you mean by removing file beat!!

Possibility-1

if you have uninstalled and installed again, then obviously file beat will read the data from the path again(which you have re-ingested and post it to logstash->elasticsearch->kibana(assuming old data is not been removed from elastic node) hence the duplicates.

Possibility-2.

You just have stopped filebeat,configured for logstash and restarted filebeat and may be your registry file is not been updated properly during shutdown(as you know,file beat reads line by line and update the registry file upto what line it has successfully published to logstash/elasticsearch/kafka etc and if any of those output servers face any difficulty processing huge load of input coming from filebeat then filebeat waits until those servers are available for further processing of input data.Once those output servers are available,filebeat reads the registry file and scan upto what line it has published and starts publishing next line onwards).

Sample registry file will be like

{
"source": "/var/log/sample/sample.log",
"offset": 88,
"FileStateOS": {
    "inode": 243271678,
    "device": 51714
},
"timestamp": "2017-02-03T06:22:36.688837822-05:00",
"ttl": -2
}

As you can see, it maintains timestamp in the registry file. So this is one of the reasons for duplicates.

For further references, you can follow below links

https://discuss.elastic.co/t/filebeat-sending-old-logs-on-restart/46189 https://discuss.elastic.co/t/deleting-filebeat-registry-file/46112

https://discuss.elastic.co/t/filebeat-stop-cleaning-registry/58902

Hope that helps.