Logstash : Mutate filter does not work

Question

I have the following filter

filter {
    grok {
    break_on_match => false
        match => { 'message' => '\[(?<log_time>\d{0,2}\/\d{0,2}\/\d{2} \d{2}:\d{2}:\d{2}:\d{3} [A-Z]{3})\]%{SPACE}%{BASE16NUM}%{SPACE}%{WORD:system_stat}%{GREEDYDATA}\]%{SPACE}%{LOGLEVEL}%{SPACE}(?<log_method>[a-zA-Z\.]+)%{SPACE}-%{SPACE}%{GREEDYDATA:log_message}%{SPACE}@%{SPACE}%{IP:app_host}:%{INT:app_port};%{SPACE}%{GREEDYDATA}Host:%{IPORHOST:host_name}:%{POSINT:host_port}' }
    match => { 'message' => '\[(?<log_time>\d{0,2}\/\d{0,2}\/\d{2} \d{2}:\d{2}:\d{2}:\d{3} [A-Z]{3})\]'}
    }
    kv{
      field_split => "\n;"
      value_split => "=:" 
      trimkey => "<>\[\],;\n"
      trim => "<>\[\],;\n"
    }
    date{
    match => [ "log_time","MM/dd/YY HH:mm:ss:SSS z" ]
    target => "log_time"
    locale => "en"
    }

    mutate {

      convert => { 
              "line_number" => "integer" 
              "app_port" => "integer" 
              "host_port" => "integer" 
              "et" => "integer" 

      }

      #remove_field => [ "message" ]

    }

    mutate {

    rename => {

              "et" => "execution_time"
              "URI" => "uri"
              "Method" => "method"

      }
    }

}

i can get results out of the grok and kv filters but neither of the mutate filters work. Is it because of the kv filter?

EDIT: Purpose

my problem is that my log contains heterogenous log records. For example

[9/13/16 15:01:18:301 EDT] 89798797 SystemErr                                                     jbhsdbhbdv [vjnwnvurnuvuv] INFO djsbbdyebycbe - Filter.doFilter(..) took 0 ms.

[9/13/16 15:01:18:302 EDT] 4353453443 SystemErr                                                    sdgegrebrb [dbebtrntn] INFO sverbrebtnnrb - [SECURITY AUDIT] Received request from: "null" @ wrvrbtbtbtf:000222; Headers=Host:vervreertherg:1111
Connection:keep-alive
User-Agent:Mozilla/5.0
Accept:text/css,*/*;q=0.1
Referer:https:kokokfuwnvuwnev/ikvdwninirnv/inwengi
Accept-Encoding:gzip
Accept-Language:en-US,en;q=0.8
; Body=; Method=GET; URI=dasd/wgomnwiregnm/iwenviewn; et=10ms; SC=200

all i care about is capturing the timestamp at the beginning of each record and a few other fields if they are present. i want Method,et,Host,loglevel and URI. If these fields are not present, i still want to capture the event with the loglevel and the message being logged.

is it advisable to capture such events using the same logstash process? should i be running two logstash processes? The problem is that i dont know the structure of the logs beforehand, apart from the few fields that i do want to capture.

Multiline config

path => ["path to log"]
start_position => "beginning" 
ignore_older => 0 
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^\[\d{0,2}\/\d{0,2}\/\d{2} \d{2}:\d{2}:\d{2}:\d{3} [A-Z]{3}\]"
negate => "true"
what => "previous"

If you could add log lines and the expected output, it would help people understand your problem, As it is, I don't understand what your problem is. — baudsp

fylie fylie · Accepted Answer · 2016-09-19T23:29:39

Maybe it is because some fields (line_number, et, URI, Method) aren't being created during the initial grok. For example, I see you define "log_method" but in mutate->rename, you refer to "Method". Is there a json codec or something applied in the input block that adds these extra fields?

If you post sample logs, I can test them with your filter and help you more. :)

EDIT: I see that the log you sent has multiple lines. Are you using a multiline filter on input? Could you share your input block as well?

You definitely don't need to run two Logstash processes. One Logstash can take care of multiple log formats. You can use conditionals, try/catch, or mark the fields as optional by adding a '?' after.

MORE EDIT: I'm getting output that implies that your mutate filters work:

"execution_time" => 10,
"uri" => "dasd/wgomnwiregnm/iwenviewn",
"method" => "GET"

once I changed trimkey => "<>\[\],;\n" to trimkey => "<>\[\],;( )?\n". I noticed that those fields (et, Method) were being prefixed with a space.

Note: I'm using the following multiline filter for testing, if yours is different it would affect the outcome. Let me know if that helps.

codec => multiline {
  pattern => "\n"
  negate => true
  what => previous
}

Logstash : Mutate filter does not work

1 Answers