Trucating Logstash message field with gsub

2
votes

I am attempting to truncate my messages coming in through my logstash forwarder. My data is ok to truncate and without truncation it is being cut off in my log provider causing json breakage.

I had added the following config to my logstash config

filter {
  mutate {
    rename => { "message" => "@message" }
  }
  mutate {
    gsub => ["@message", "^.{1000}(.*)$", "..."]
  }
}

My assumption is that gsub regex will get from the 1001st character till end of field value and replace it with "..."

I also had to rename the field message to @message to keep it consistent.

But it does not seem to want to trim at all. This is with logstash 2.3.

Any ideas of what I may be doing wrong or another filter type I should be using to do truncation?

3
logginglogstash
Hi @MechaStorm if the answer has solved your question please consider accepting it. This indicates to the wider community that you've found a solution. But there is no obligation to do this. - baudsp

3 Answers

3
votes

You can do it with the ruby filter:

ruby {
    code => "event['@message'] = event['@message'][0...1000]+'...'"
  }

This will replace the @message field with the first 1000 characters from the @message field, followed by ....

Update: it might not work on all versions of Logstash. It was tested on the 2.4 version (I think) and will not work on more recent version of Logstash; the other solutions using gsub should work across all versions.

1
votes

This worked for me:

    mutate {
        gsub => [             
            "message", "(^.{1,1000}).*$", "\1..."
        ]
    }

I originally tried the ruby filter suggestion from @baudsp, but couldn't get it to work.

1
votes

After 5.1 logstash has a truncate filter: https://www.elastic.co/guide/en/logstash/5.1/plugins-filters-truncate.html

truncate {
  length_bytes => ...
}

Before that I think the best alternative is mutate with a range filter:

range {
    ranges => [ "message", 1000, 100000, "tag:longmessage"]
  }

if "longmessage" in [tags] {
  mutate {
    #truncate long messages
    gsub => [ "message", "(^.{1,1000}).*$", "\1..."]
  }
}

The previous answer works but set 3 dots at the end of each messages.