Losing messages between logstash and elasticsearch

1
votes

I'm set up as follows while we're testing out the ESK stack and learning how all this works:

  • Client server (CS) running filebeat sends output to logstash running on site aggregator node (AN).

  • AN runs logstash with some filters and then forwards to rabbitmq on our Elasticsearch/Kibana node (ESK).

  • ESK runs rabbitmq, and logstash pulls messages from rabbitmq before sending output to elasticsearch (without filtering). Kibana is our visualization (obviously) and we're all pretty new with elasticsearch so we're not doing much with it directly.

Here's the problem:

CS generates a message. It definitely gets sent to AN, where logstash filters it and fowrwards it on (after echoing it to logstash.stdout). The logstash instance on ESK also sees it (and writes it to logstash.stdout). I can see the messages in both logstash instances. They match and are appropriately tagged. But they aren't visible in Kibana.

Our configs and a sample message from both logs are all in gist form here: https://gist.github.com/wortmanb/ebd37b8bea278d06ffa058c1513ef940

Where could these messages be going? They're not showing up in Kibana -- if I filter on messages with tags: "puppet", I get basically nothing during timeframes when I know these messages are flowing.

Any debugging suggestions?

1
elasticsearchlogstashkibana
Do you see your logs in the Elasticsearch ? Can you put your hand on a log that was not indexed in the Elasticsearch? – Erik
Yes - the puppetserver log messages aren't getting in there. Well, more to the point, the ones that are getting tagged with "_grokparsefailure" are getting in. The ones I'm successfully parsing aren't. We get the occasional multiline stackdump which fails to parse -- I haven't dealt with those yet. I found these by searching for source: "/var/log/puppetlabs/puppetserver/puppetserver.log" in Kibana. – Bret
Just on your architecture, do you really need rabbitmq? You could just transfer your log directly from the logstash at the aggregator node (AN) to ES at your Elasticsearch/Kibana node (ESK). Or perhaps there are constraints not detailed here that force this architecture. – baudsp

1 Answers

1
votes

The problem is that you are parsing the date of the log with the date filter, which, by default, replace the @timestamp field, which is used to filter according to the date.

I get basically nothing during timeframes when I know these messages are flowing.

So the messages are not in the timeframe during which they where flowing, but during the timeframe they were written.

You can see the "_grokparsefailure" logs since their date is not parsed, then the @timestamp is the reception date in Logstash.

So you'll to change your timeframe to one including the dates of your logs.