How to process multilines in logstash with multiple worker threads?

Question

I would like to process mulitline logs with logstash using multiple worker threads for performance but multiline filter doesn't work: - https://github.com/elastic/logstash/pull/1591 - https://github.com/elastic/logstash/issues/1590

solutions for now:

• using multiple logstash-forwarder and send them to different lumberjack port (scales very poorly: new logstash-forwarder for each logfile that has multilines)
• using an extra logstash-receiver with redis output and an extra logstash indexer that reads from redis and uses one worker thread (https://groups.google.com/forum/#!msg/logstash-users/fUhulxmZ0Ek/HfGev7BOhCUJ)
• as a hack, I tried to put a type check that can have multiline in logstash output part with lumberjack output and resending them to the same logstash instance's different lumberjack input (different port) that has codec multiline but it doesn't process them
• I could also use rabbitMQ that reads events but then I should merge multilines before they sent to rabbitMQ (How should I do that?)
• rsyslog can also send logs but it needs special format of multiline logs and doesn't work with regexp (https://serverfault.com/questions/622035/how-do-i-configure-rsyslog-to-deal-with-mysql-slow-query-log-multi-line-messages)

Clearly none of these is nice, so is there anyone who has a decent solution for this?

Answer 1

An alternative would be using log-courier instead of logstash-forwarder. It performs the multiline prior sending it to logstash:

https://github.com/driskell/log-courier/blob/master/docs/codecs/Multiline.md

There is a request to support this also in logstash-forwarder:

https://github.com/elastic/logstash-forwarder/issues/309

I'm curious, how did you do to send multiline logs to a specific single threaded logstash and the rest to the others?

Answer 2

There's another alternative:

1. Do your multiline processing directly in your inputs, using the multiline codec. Obviously this bit will be single-threaded per input.

2. Then use the rest of your filters normally. This will be multi-threaded, per filter worker, if -w <threads> was provided.

The Logstash Pipeline's Thread Model is documented as follows:

The thread model in Logstash is currently:

input threads | filter worker threads | output worker

Answer 3

I ended up using a separete logstash to process multilines. It is configured to use one worker thread and uses the multiline filter. If the performance is still not okay, you can add more of logstash that uses just one worker thread. Then when someone fixes it we will remove this extra logstash instance.

Answer 4

Try something like,

 multiline {
        pattern => "(Stack trace:)|(^#.+)|(^\"\")|(  thrown+)|(^\s)"
        what    => "previous"
      }

I'm using it for Php Fatal errors.