Defining a grok filter over multiple lines in logstash

Question

I have a very long logstash grok filter:

match => { 'message' => '%{MONTH:month} %{NUMBER:day} %{TIME:time} %{WORD:log_host} %{WORD:generator}\[%{NUMBER:unknown}\]: %{IP:connIP}:%{NUMBER:connPort} \[.*\] %{WORD:namespace}\~? %{NOTSPACE:unknown} %{NOTSPACE:unknown} %{NUMBER:res_statuscode} %{NUMBER:unknown} (?<unknown>\-.*\-) %{NOTSPACE:unknown} %{NOTSPACE:unknown} \"%{WORD:method} %{PATH:path} %{DATA:httpversion}\"' }

Any way to break this up over multiple lines? I tried the following:

match => { 'message' => '%{MONTH:month} %{NUMBER:day} %{TIME:time} %{WORD:log_host}'
                                ' %{WORD:generator}\[%{NUMBER:unknown}\]: %{IP:connIP}:%{NUMBER:connPort}'
                                ' \[.*\] %{WORD:namespace}\~? %{NOTSPACE:unknown} %{NOTSPACE:unknown}'
                                ' %{NUMBER:res_statuscode} %{NUMBER:unknown} (?<unknown>\-.*\-) %{NOTSPACE:unknown}'
                                ' %{NOTSPACE:unknown} \"%{WORD:method} %{PATH:path} %{DATA:httpversion}\"' }

But it's giving me errors, even separating the strings with commas doesn't work: {:timestamp=>"2016-09-30T08:38:50.549000+0000", :message=>"fetched an invalid config" ...

There's no mention of handling this in the official documentation:

Another solution would be to split the grok pattern in two, with the first filter leaving the end of the message to be parsed by the second filter. But watch out for your perf (check this blog post). — baudsp
@baudsp (1) It's supposed to be a solution that improves visibility for my team regardless of how everyone has set up their editor, so this solution won't do. (2) It's not an ideal solution as I really want to match against the whole pattern (and fail if it doesn't match) but I'll keep it in mind if I need to pipe it like this for added visibility. — Petrus K.
I understand, the second solution is not really good. (and the first one more of a joke, I should have added ;)). Another idea would be to use a custom pattern file (cf doc), which could be easier to read, if you use a few patterns and use them instead in your grok filter — baudsp
I'm only using one custom pattern, otherwise I'm reusing all the natively supported patterns. I think substituting my grok match filter would only hide the visibility of my pattern from the team, but I'll consider using it if there are several input formats that need to be matched. Thanks for the tip. — Petrus K.

Soundarya Thiagarajan Soundarya Thiagarajan · Accepted Answer · 2016-11-10T07:45:03

The error msg clearly shows there is an error with the config file.

this link may help : https://discuss.elastic.co/t/grok-multiple-match-logstash/27870

   match => {"message" => ["(%{EXIM_DATE:exim_date} )(%{EXIM_MSGID:exim_msg_id} )(?<msg_c>Completed)",
                            "(%{EXIM_DATE:exim_date} )(%{EXIM_MSGID:exim_msg_id} )(?<msg_f>frozen)"
                           ]
             }

Defining a grok filter over multiple lines in logstash

1 Answers