Logstash mutate filter on json data

Question

I'm tying to use logstash to perform a transform on json data that is being logged to a text file.

Logstash config snippet:

filter {
  mutate {
    gsub => [
      # replace all '::~~' with '\r\n'
      "alert.explanation.cnc-services.cnc-service.channel", "\:\:~~", "\\r\\n"
#      "channel", "~~", "n"
    ]
  }
}

Input json:

{
  "alert": {
    "explanation": {
      "cnc-services": {
        `enter code here`"cnc-service": {
          "channel": "TEXT1 TEXT2::~~TEXT3::~~TEXT4::~~TEXT5::~~TEXT6::~~::~~"
        }
      }
    }
  }
}

Desired output:

{
  "alert": {
    "explanation": {
      "cnc-services": {
        "cnc-service": {
          "channel": "TEXT1 TEXT2\r\nTEXT3\r\nTEXT4\r\nTEXT5\r\nTEXT6\r\n\r\n"
        }
      }
    }
  }
}

The input data is only a portion of the whole json document, there are may other fields but I only need to modify this one. This data is generated by another application and it can't be modified. None of the articles I've seen for logstash talk about doing something like this.

edit: Here is the input section of the logstash config

input {
    # Read from log file
    file {
        codec => "json"
        type => "fireeye"
        path => ["/var/log/app/events.json"]
    }
}

Pedantic but important question: as Logstash receives this data, is it parsed into event fields and subfields, or is the JSON a string in one flat field? — rutter
I believe it is parsing it as json. Here's the input section: input { # Read from log file file { codec => "json" type => "fireeye" path => ["/var/log/app/events.json"] } } — nyc_edwardd
Hmm. Documentation on the codecs is pretty hard to find, unfortunately. I've had a little bit of luck reading line-by-line with the file input, then parsing a single field into subfields using the json filter plugin. I'd suggest asking on the logstash-users list if you can't find help here. — rutter

mrucci mrucci · Accepted Answer · 2014-09-24T09:35:28

Some trial and error showed that the key in the gsub array should have the format.

first_level[second_level][third_level]...

So in your case:

filter {
  mutate {
    gsub => ["alert[explanation][cnc-services][cnc-service][channel]", "whatever"]
  }
}

I don't know exactly what's the logic behind it, but it works.

Logstash mutate filter on json data

1 Answers