Filebeat 6.1.0 and exclude_lines option

3
votes

I'm having some issues getting filebeat to exclude lines from apache2's access log. I've got the apache2.yml config enabled and it does exclude log files but not lines. Here's an example of the line I'm looking to exclude:

example.site.com:80 192.168.0.1 - - [20/Dec/2017:10:18:37 -0500] "GET /server-status?auto= HTTP/1.1" 301 522 "-" "Go-http-client/1.1"

Here are the ways I've tried to use the regexp format with the exclude_lines: option in the apache2.yml file:

exclude_lines: ['.*server-status.*'] exclude_lines: ['(?i:/server-status\?auto=)'] exclude_lines: ['GET /server-status']

as well as a few other variations. I've also tried a processor in both the main filebeat config file and in the apache2 module config like this:

- module: apache2 # Access logs access: enabled: true processors: - drop_event: when: regexp: apache2.access.url: "server-status"

I tried it with contains: in place of regexp: but that doesn't seem to work either. But this:

exclude_files: [".gz$"] DOES work; from the log file:

017-12-21T10:15:27-05:00 DBG [prospector] Exclude file: /var/log/apache2/example.site.com/access.log.10.gz

It's all been making me a little nuts. Doesn't matter which way I try the expression or processor, the example log line above is still being fed to elasticsearch. I posted a similar question over at the discuss.elastic.co forums but no one has replied thus far.

Does anyone have experience excluding log lines? Filebeat, Elasticsearch, and Logstash are all at version 6.1.0.

Thanks in advance!

1
elasticsearchapache2filebeat
Try this exclude_lines: ['.*server-status.*','(?i:/server-status\?auto=)','GET /server-status'] in filebeat.yml – Ali Ahmad
Thanks for the reply. Unfortunately, even after adding you suggesting into the filebeat.yml config file and restarting the service, those events are still being passed through into ES. – robscott27
I almost get the feeling that it's not even reading/using the exclude_lines option in the first place. I decided to try to drop everything in the log file, regardless of the content using ['.'] but everything still goes into ES. – robscott27
I figured out what the issue was. I noticed that even though I had commented out the exclude_files line, it was still excluding the .gz log files, even though it shouldn't. So I went digging. Turns out, there's an access.yml file located here: /usr/share/filebeat/module/apache2/access/config which still had the exclude_files option set to ignore the .gz files. Below that line, I added: exclude_lines: ['.*(?:server-status).*'] and restarted the service. Watched the log file, and bingo, lines are being dropped now. – robscott27

1 Answers

0
votes

I figured out what the issue was. I noticed that even though I had commented out the exclude_files line, it was still excluding the .gz log files, even though it shouldn't. So I went digging.

Turns out, there's an access.yml file located here: /usr/share/filebeat/module/apache2/access/config which still had the exclude_files option set to ignore the .gz files. Below that line, I added: exclude_lines: ['.*(?:server-status).*'] and restarted the service. Watched the log file, and bingo, lines are being dropped now.

-- robscott27