using grok to extract log data

Question

i am trying to extract data from the log file using grok.my log lines looks like this.

[Server 192.178.35.40] testweb.de 63.239.73.83 - - [19/Nov/2017:23:27:26 +0100] \"GET /service/want/teaser2/Buk/ HTTP/1.1\" 200 319 \"-\" \"https://testweb.de/Suche/Buk/Bonn\" \"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\" \"65.259.77.67\" 0

i am expecting something like this

server : 192.178.35.40
website : testweb.de
clientip : 63.239.73.83
timestamp:19/Nov/2017:23:27:26 +0100
method:GET
RESOURCE:/service/want/teaser2/Buk/ HTTP/1.1
RESPONCE:200
TIMETAKEN:319
USERAGENT:Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile 
Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
COOKIE:0

tried on https://grokdebug.herokuapp.com/ by giving pattern,

 %{ip:SERVER}

and received the results but unable to parse the remaining data

Sufiyan Ghori Sufiyan Ghori · Accepted Answer · 2018-06-15T01:10:12

How did you expect to extract everything in their own fields with just a single pattern to match?

You need to match every field separately in order to get your desired output. Can you try this?

%{IPV4:server}\] %{HOSTNAME:website} %{IPV4:client} - - \[%{HTTPDATE:timestamp}\] \\"%{WORD:method} (?<resource>%{NOTSPACE} HTTP/%{NUMBER})\\" %{NUMBER:response} %{NUMBER:TimeTaken} \\"-\\" \\"%{URI}\\" \\"%{GREEDYDATA:useragent}\).*%{NUMBER:cookie}

This will output,

{
  "server": [
    [
      "192.178.35.40"
    ]
  ],
  "website": [
    [
      "testweb.de"
    ]
  ],
  "client": [
    [
      "63.239.73.83"
    ]
  ],
  "timestamp": [
    [
      "19/Nov/2017:23:27:26 +0100"
    ]
  ],
  "MONTHDAY": [
    [
      "19"
    ]
  ],
  "MONTH": [
    [
      "Nov"
    ]
  ],
  "YEAR": [
    [
      "2017"
    ]
  ],
  "TIME": [
    [
      "23:27:26"
    ]
  ],
  "HOUR": [
    [
      "23"
    ]
  ],
  "MINUTE": [
    [
      "27"
    ]
  ],
  "SECOND": [
    [
      "26"
    ]
  ],
  "INT": [
    [
      "+0100"
    ]
  ],
  "method": [
    [
      "GET"
    ]
  ],
  "resource": [
    [
      "/service/want/teaser2/Buk/ HTTP/1.1"
    ]
  ],
  "NOTSPACE": [
    [
      "/service/want/teaser2/Buk/"
    ]
  ],
  "NUMBER": [
    [
      "1.1"
    ]
  ],
  "BASE10NUM": [
    [
      "1.1",
      "200",
      "319",
      "0"
    ]
  ],
  "response": [
    [
      "200"
    ]
  ],
  "TimeTaken": [
    [
      "319"
    ]
  ],
  "URI": [
    [
      "https://testweb.de/Suche/Buk/Bonn"
    ]
  ],
  "URIPROTO": [
    [
      "https"
    ]
  ],
  "USER": [
    [
      null
    ]
  ],
  "USERNAME": [
    [
      null
    ]
  ],
  "URIHOST": [
    [
      "testweb.de"
    ]
  ],
  "IPORHOST": [
    [
      "testweb.de"
    ]
  ],
  "HOSTNAME": [
    [
      "testweb.de"
    ]
  ],
  "IP": [
    [
      null
    ]
  ],
  "IPV6": [
    [
      null
    ]
  ],
  "IPV4": [
    [
      null
    ]
  ],
  "port": [
    [
      null
    ]
  ],
  "URIPATHPARAM": [
    [
      "/Suche/Buk/Bonn"
    ]
  ],
  "URIPATH": [
    [
      "/Suche/Buk/Bonn"
    ]
  ],
  "URIPARAM": [
    [
      null
    ]
  ],
  "useragent": [
    [
      "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html"
    ]
  ],
  "cookie": [
    [
      "0"
    ]
  ]
}

using grok to extract log data

1 Answers