logstash parsing apache drupal logs

Question

I have this log:

[Fri May 15 01:20:48.661420 2020] [proxy_fcgi:error] [pid 11943:tid 139947462346496] [client 154.15.19.140:64717] AH01071: Got error 'PHP message: Company Care|https://webdemo2.nics.com|severity=error|type=feature_card_manager|ip=154.15.19.140|uri=https://webdemo2nics.com/ca/user/payment-information|referer=https://webdemo2.nics.com/ca/user/payment-information|uid=68055|link=|message=Failed to add card. HTTP Response: {"code":71,"category":1,"message":"Declined","reference":""}', referer: https://webdemo2.nics.com/ca/user/payment-information

and I'm trying to send it via logstash. Here is my logstash config:

input {
    file {
      path => "/tmp/test.log"
    }
}
filter {
  date {
    match => [ "logdate", "E MMM dd HH:mm:ss.SSS yyyy" ]
    target => "@timestamp"
  }
  grok {
        match => { "message" => "\[(?<timestamp>%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\]\s+\[%{WORD}:%{WORD:loglevel}\].*\[client %{URIHOST:client}\]\s++%{WORD:error_code}\:.*\|type=%{WORD:type}\|.*\|uri=%{URIPATH:uri}\|uid=%{NUMBER:uid}\|.*%{GREEDYDATA:message}$"
                 }
        overwrite => ["message"]
  }
  mutate{
      remove_field => "@version"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

The problem is that it is not overwriting @timestamp with timestamp. I tried using mutate rename but that just made me lose the field. Does anyone know how to change the config file to fix the timestamp issue? Also the timestamp appears to be parsed as string rather than a date object. Is it supposed to come from my date pattern or a combination of date pattern and grok pattern?

Here is the output:

{
        "client" => "154.15.19.140:64717",
    "error_code" => "AH01071",
    "@timestamp" => 2020-07-15T16:58:52.916Z,
      "loglevel" => "error",
          "host" => "kourosh-VirtualBox",
          "port" => "64717",
          "path" => "/tmp/test.log",
     "timestamp" => "Fri May 15 01:20:48.661420 2020",
       "message" => "[Fri May 15 01:20:48.661420 2020] [proxy_fcgi:error] [pid 11943:tid 139947462346496] [client 154.15.19.140:64717] AH01071: Got error 'PHP message: Company|https://webdemo2.nics.com|severity=error|type=feature_card_manager|ip=154.5.59.140|uri=https://webdemo2.nics.com/ca/user/payment-information|referer=https://webdemo2.nics.com/ca/user/payment-information|uid=55|link=|message=Failed to add card. HTTP Response: {\"code\":71,\"category\":1,\"message\":\"Declined\",\"reference\":\"\"}', referer: https://webdemo2.nics.com/ca/user/payment-information"
}

max max · Accepted Answer · 2020-07-21T17:35:49

I figured it out. The date section needs to come after grok section and target can be used to overwrite @timestamp. Here is the working config:

    input {
    file {
      path => "/tmp/test.log"
    }
}
filter {
  grok {
        match => { "message" => "\[(?<event_timestamp>%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\]\s+\[%{WORD}:%{WORD:loglevel}\].*\[client %{URIHOST:client}\]\s+%{WORD:error_code}\:.*\|type=%{WORD:type}\|.*\|uri\=%{GREEDYDATA:uri}\|referer.*\|uid=%{NUMBER:uid}\|.*\|message\=%{GREEDYDATA:message}(\s+HTTP Response\:%{GREEDYDATA:response}.*)?(\,\sreferer.*).*"
                 }
        overwrite => ["message"]
  }
  date {
    match => [ "event_timestamp", "E MMM dd HH:mm:ss.SSSSSS yyyy" ]
    target => "@timestamp"
    add_field => { "debug" => "timestampMatched"}
  }

  mutate{
      remove_field => ["@version", "@tags"]
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

logstash parsing apache drupal logs

1 Answers