logstash-2.2.2, windows, IIS log file format

Question

when I'm parsing iis log file in UTF-8 format I'm getting below error and When I'm parsing log file using ANSI format there is nothing working Logstash just display message on console " Logstash startup completed". There is almost 1000 files on my server i can't change each file format from ANSI to UTF-8. Can you please help where I need to change in my config file. I'm also attaching debug file when I'm parsing files on UTF-8 format. I'm using elastic search on same box and its completely working fine. I'm also able to telnet port 9200 with 127.0.0.1.

Log sample:

2016-03-26T05:40:40.764Z WIN-AK44913P759 2016-03-24 00:16:31 W3SVC20 ODSANDBOXWEB01 172.x.x.x GET /healthmonitor.axd - 80 - 172.x.x.x HTTP/1.1 - - - www.xyz.net 200 0 0 4698 122 531

stdout output:

{
  "message" => "2016-03-24 04:43:02 W3SVC20 ODSANDBOXWEB01 172.x.x.x GET /healthmonitor.axd - 80 - 172.x.x.x HTTP/1.1 - - - www.xyz.net 200 0 0 4698 122 703\r",
  "@version" => "1",
  "@timestamp" => "2016-03-26T05:42:15.045Z",
  "path" => "C:\\IISLogs/u_ex160324.log",
  "host" => "WIN-AK44913P759",
  "type" => "IISLog",
  "tags" => [
     [0] "_grokparsefailure"
  ]
}

Below is my logstash conf file configuration

input {
  file {
    type => "IISLog"
    path => "C:\IISLogs/u_ex*.log"
    start_position => "beginning"
  }
}
filter {
  #ignore log comments
  if [message] =~ "^#" {
    drop {}
  }
  grok {
    match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytes:int} %{NUMBER:timetaken:int}"]
  }
  #Set the Event Timesteamp from the log
  date {
    match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
    timezone => "Etc/UCT"
  }
  useragent {
    source=> "useragent"
    prefix=> "browser"
  }
  mutate {
    remove_field => [ "log_timestamp"]
  }
}
# output logs to console and to elasticsearch
output {
  stdout {}
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
  }
  stdout {
    codec => rubydebug
  }
}

Alain Collins Alain Collins · Accepted Answer · 2016-03-26T15:19:22

The _grokparsefailure tag means that your grok pattern didn't match your input. It looks like you're intending the pattern to skip the first two fields, which is fine.

Then, looking at the next four fields, I see:

2016-03-24 00:16:31 W3SVC20 ODSANDBOXWEB01 172.1.1.1

but your pattern is looking for

%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{WORD:method}

You haven't accounted for the IP address (since ODSANDBOXWEB01 is going into [site]).

Building grok patterns is a deliberate, iterative process. Start at the debugger. Enter a sample input line and then add grok patterns - one at a time! - until the entire line has been matched.

Also, when you obfuscate your data, please leave it as valid data. Changing the ip to 172.x.x.x means that it won't match the %{IP} pattern without us having to figure out what you did. I changed it to 172.1.1.1 in this example.

logstash-2.2.2, windows, IIS log file format

1 Answers