Create a Kibana graph from logstash logs

Question

I need to create a graph in kibana according to a specific value.

Here is my raw log from logstash :

2016-03-14T15:01:21.061Z Accueil-PC 14-03-2016 16:01:19.926 [pool-3-thread-1] INFO com.github.vspiewak.loggenerator.SearchRequest - id=300,ip=84.102.53.31,brand=Apple,name=iPhone 5S,model=iPhone 5S - Gris sideral - Disque 64Go,category=Mobile,color=Gris sideral,options=Disque 64Go,price=899.0

In this log line, I have the id information "id=300". In order to create graphics in Kibana using the id value, I want a new field. So I have a specific grok configuration :

grok {
 match => ["message", "(?<mycustomnewfield>id=%{INT}+)"]        
}

With this transformation I get the following JSON :

{
"_index": "metrics-2016.03.14",
"_type": "logs",
"_id": "AVN1k-cJcXxORIbORG7w",
"_score": null,
"_source": {
  "message": "{\"message\":\"14-03-2016 15:42:18.739 [pool-1950-thread-1] INFO com.github.vspiewak.loggenerator.SellRequest - id=300,ip=54.226.24.77,[email protected],sex=F,brand=Apple,name=iPad R\\\\xE9tina,model=iPad R\\\\xE9tina - Noir,category=Tablette,color=Noir,price=509.0\\\\r\",\"@version\":\"1\",\"@timestamp\":\"2016-03-14T14:42:19.040Z\",\"path\":\"D:\\\\LogStash\\\\logstash-2.2.2\\\\logstash-2.2.2\\\\bin\\\\logs.logs.txt\",\"host\":\"Accueil-PC\",\"type\":\"metrics-type\",\"mycustomnewfield\":\"300\"}",
 "@version": "1",
 "@timestamp": "2016-03-14T14:42:19.803Z",
 "host": "127.0.0.1",
 "port": 57867
},
"fields": {
 "@timestamp": [
  1457966539803
 ]
},
"sort": [
 1457966539803
]}

A new field was actually created (the field 'mycustomnewfield') but within the message field ! As a result I can't see it in kibana when I try to create a graph. I tried to create a "scripted field" in Kibana but only numeric field can be accessed.

Should I create an index in elasticSearch with a specific mapping to create a new field ?

Feels like something else is going on with your config. Though it doesn't answer your question, I would take the stuff from "id=" to the end and run it through the kv{} filter. — Alain Collins
Thank you for the answer, I tried with the kv filter and it does the same thing => The field is always within the message field. I will use the kv filter but I still have a problem. — Santana6.35
Like I said, it doesn't feel like that grok{} line is the one screwing you up. I would look through the whole config for other things going on with [message]. — Alain Collins
So there is actually something wrong ? The new field should not have been inserted within the "message" field ? — Santana6.35

Santana6.35 Santana6.35 · Accepted Answer · 2016-03-15T10:12:44

There was actually something wrong with my configuration. I should have paste the whole configuration with my question. In fact i'm using logstash as a shipper and also as a log server. On the server side, I modified the configuration :

input {
tcp {
    port => "yyyy"
    host => "x.x.x.x"
    mode => "server"
    codec => json # I forgot this option
}}

Because the logstash shipper is actually sending json, I need to advice the server about this. Now I no longer have a message field within a message field, and my new field is inserted at the right place.

Create a Kibana graph from logstash logs

1 Answers