Logstash custom date format and irregular spaces

Question

Receiving a parsing failure with my grok match. I can't seem to find anything that will match my log.

Here is my log:

2016-06-14 14:03:42 1.1.1.1 GET /origin-www.site.com/ScriptResource.axd?d= jEHA4v5Z26oA-nbsKDVsBINPydW0esbNCScJdD-RX5iFGr6qqeyJ69OnKDoJgTsDcnI1&t=5f9d5645 200 26222 0 "http://site/ layouts/CategoryPage.aspx?dsNav=N:10014" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G318HZ Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.95 Mobile Safari/537.36" "cookie"

Here is my grok match. It works fine in the grok debugger.

filter {
  grok {
    match => { 'message' => '%{DATE:date} %{TIME:time} %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:status} %{NUMBER:bytes} %{NUMBER:time_taken} %{QUOTEDSTRING:referrer} %{QUOTEDSTRING:user_agent} %{QUOTEDSTRING:cookie}' }
  }
}

EDIT: I decided to do a screenshot of what my log file looks like as the spaces dont come over when copying and pasting. Those appear to be single spaces when I copy/paste.

The logline you posted has a space in the URI params at xd?d= jEHA4 — Will Barnwell

Will Barnwell Will Barnwell · Accepted Answer · 2016-06-29T22:18:23

Beside the space in that logline you posted, which I assume won't exist in your logs, your pattern is incorrect on the date parsing. Logstash DATE follows this pattern:

DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
DATE %{DATE_US}|%{DATE_EU}

Which doesn't match your YYYY-MM-dd format. I recommend using a pattern file and defining a custom date format

CUST_DATE %{YEAR}-%{MONTHNUM2}-%{MONTHDAY}

then your pattern can be

%{CUST_DATE:date} %{TIME:time} %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:status} %{NUMBER:bytes} %{NUMBER:time_taken} %{QUOTEDSTRING:referrer} %{QUOTEDSTRING:user_agent} %{QUOTEDSTRING:cookie}

EDIT: You may be able to handle weird whitespace with a gsub, this won't remove whitespace, but will normalize spaces to all be 1 " "

mutate {
    gsub => [
      # replace all whitespace characters or multiple adjacent whitespace characters with one space
      "message", "\s+", " "
    ]
  }

Logstash custom date format and irregular spaces

1 Answers