Logstash filter - half json line parse

Question

I'm using 'filebeat' as a shipper an the client send it to redis, read from redis with logstash and send it to ES.

I'm trying to parse the following example line:

09:24:01.969 watchdog - INFO - 100.140.2 PASSED: Mobile:Mobile[].popover["mc1814"].select(2,) :706<<<<<<<<<<<<<<<<<<< {"actionDuration":613}

In the end I want to have a field names: "actionDuration" with the value: 613.

As you can see it's partially json. - I've tried to use grok filter, with add_field and match and I've tried to change a few configurations in the filebeat and logstash.

I'm using the basic configurations: filebeat.conf:

filebeat.prospectors:

input_type: log

paths:

/sketch/workspace/sanity-dev-kennel/out/*.log

fields:

type: watchdog

BUILD_ID: 82161

If there's a possibility to do it in the filebeat side I prefer, but it's also good in the Logstash side.

Thanks a lot, Moshe

sysadmin1138 sysadmin1138 · Accepted Answer · 2017-03-28T12:32:17

This sort of partial-formatting is best handled on the Logstash side, not the shipper. The filters/transforms available in FileBeat aren't up to that. A Logstash filter pipeline is, though.

filter {
  grok {
    match => {
      "message" => [ "(?<plain_prefix>^.*?) (?<json_segment>{.*$)"]
    }
  }
  json {
    source => "json_segment"
  }
  mutate {
    remove_field => [ "json_segment" ]
  }
}

This basic example will split your incoming message into two fields. a plain_prefix and a json_segment. The json{} filter is then used to parse the JSON data into the event. Finally, a mutate {} filter is used to remove the json_segment field from the event, as it has already been parsed and included.

Note: the .*? in the plain_prefix is critical in this filter. Constructed this way, everything from the first { onward is considered part of the JSON segment. If you use .*, the JSON segment will be everything from the last {, which will be a problem with complex JSON datastructures.

Logstash filter - half json line parse

1 Answers