I have this kind of log 21.4.1.2 - - [28/Dec/2016:12:18:40 +0000] "GET a/b/c/d/e/f HTTP/1.1" 200 984072 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" 0.104 0.103 . Now how should I extract this using grok pattern ? I dont know the no of fields also i.e. rest api can be a/b/c also and a/b/c/d/e/f/g also. How should I handle it so that I can group by a,b or c in kibana.
1
votes
This answer might help: stackoverflow.com/questions/37605189/…
– Val
Thanks @Val for your time but the question link which you posted is when there is query parameter but in mine case it is a part of url ex:- facebook.com/user1, facebook.com/user2 and I want to grab facebook.com as one field and user as other field. Only the twist is that there can be multiple part of url like a/b/c.
– agrawal1084
2 Answers
1
votes
If there's a known depth, you could re-grok the URL field into those fields.
If there's an arbitary depth, mutate-split could make an array of them, but they wouldn't be useful.
How about the csv{} filter, which could take "/" as the separator and would produce you a bunch of fields called "column1", "column2", etc?
0
votes
There is a %{GREEDYDATA:value} grok template that you can use to extract the API path part, from there you could split on "/". This tool can be useful when debugging grok patterns http://grokdebug.herokuapp.com/.
So start with:
%{IP:clientip} \- \- \[%{NOTSPACE:date} \+%{INT}\] \"%{WORD:action} %{GREEDYDATA:api} %{WORD:protocol}/%{NUMBER:protocolNum}\" %{NUMBER:status} %{NUMBER} %{QUOTEDSTRING} %{NUMBER} %{NUMBER}
Which will give you the api path in the api field.
Alternatively, we are working on Moesif which is an API debug and analytics tool (https://www.moesif.com/features) which may be helpful for you depending on what you require. (Full disclosure, I am the CEO)
