Logstash to kibana multiline not working for custom message

Question

I am trying push my log file through logstash to elasticsearch and display it on kibana. It works fine for the single line log records. However, it fails when it comes to the multiline filter.
Here is my sample multiline log input:

2016-06-02T04:02:29,720 INFO  Thread-25-match-entity-bolt a52488cc-316b-402e-af58-3b8a663cd76a STDIO invoke Error processing message:{
  "eid": "f9f16541-4fab-4131-a82e-e3ddf6fcd949",
  "entityInfo": {
    "entityType": "style",
    "defaultLocale": "en-US"
  },
  "systemInfo": {
    "tenantId": "t1"
  },
  "attributesInfo": {
    "externalId": 1514,
    "attributesRead": {
      "IsEntityVariantsValid": false,
      "IsEntityExtensionsValid": false
    },
    "attributesUpdated": {
      "DateAttribute": "2016-06-01T00:00:00.0000000",
      "IsEntitySelfValid": true,
      "IsEntityMetaDataValid": true,
      "IsEntityCommonAttributesValid": true,
      "IsEntityCategoryAttributesValid": true,
      "IsEntityRelationshipsValid": true
    }
  },
  "jsAttributesInfo": {
    "jsRelationship": {
      "entityId": "CottonMaterial001",
      "parentEntityId": "Apparel",
      "category": "Apparel",
      "categoryName": "Apparel",
      "categoryPath": "Apparel",
      "categoryNamePath": "Apparel",
      "variant": "1514",
      "variantPath": "1035/1514",
      "container": "Demo Master",
      "containerName": "Demo Master",
      "containerPath": "DemoOrg/Demo Master/Apparel",
      "organization": "DemoOrg",
      "segment": "A"
    },
    "jsChangeContext": {
      "entityAction": "update",
      "user": "cfadmin",
      "changeAgent": "EntityEditor.aspx",
      "changeAgentType": "PIM",
      "changeInterface": "Entity",
      "sourceTimestamp": "2016-06-01T19:48:19.4162475+05:30",
      "ingestTimestamp": "2016-06-01T19:48:19.4162475+05:30"
    }
  }
}

I have tried these logstash configs so far:

input {
  file {
    path => "path_to_logs/logs.log"
    start_position => "beginning"
  }
}
filter{
    multiline {
      negate => "true"
      pattern => "^%{TIMESTAMP_ISO8601} "
      what => "previous"
    }
    grok{
      match => { "message" => "^%{TIMESTAMP_ISO8601:JigsawTimestamp}%{SPACE}%{LOGLEVEL:JigsawLoglevel}%{SPACE}%{HOSTNAME:ThreadName}%{SPACE}%{UUID:GUID}%{SPACE}%{JAVACLASS:JigsawClassName}%{SPACE}%{WORD:JigsawMethodName}%{SPACE}%{GREEDYDATA:JigsawLogMessage}" } 
    }
}
output {

  if "_grokparsefailure" not in [tags] {
    elasticsearch { 
      hosts => ["localhost:9200"] 
    }
  }
}

The second one:

input {
  file {
    path => "path_to_logs/logs.log"
    start_position => "beginning"
    codec => multiline {
      negate => "true"
      pattern => "^%{TIMESTAMP_ISO8601} "
      what => "previous"
    }
  }
}
filter{
    grok{
      match => { "message" => "^%{TIMESTAMP_ISO8601:JigsawTimestamp}%{SPACE}%{LOGLEVEL:JigsawLoglevel}%{SPACE}%{HOSTNAME:ThreadName}%{SPACE}%{UUID:GUID}%{SPACE}%{JAVACLASS:JigsawClassName}%{SPACE}%{WORD:JigsawMethodName}%{SPACE}%{GREEDYDATA:JigsawLogMessage}" } 
    }
}
output {

  if "_grokparsefailure" not in [tags] {
    elasticsearch { 
      hosts => ["localhost:9200"] 
    }
  }
}

I tried this pattern as well:

pattern => "^\s"

However, none of this helped. All of them got _grokparsefailure tag. I want the JSON lines to be part of a single message. Please point out the mistake in this filter.

Yuvraj Gupta Yuvraj Gupta · Accepted Answer · 2016-06-12T07:33:32

In your grok filter, there are couple of mistakes via which you are unable to see any logs.

In your sample data after INFO there are 2 spaces.
For the field JigsawClassName you are using JAVACLASS as input which is wrong for your log.

Why JAVACLASS is wrong?

It's implementation is as :-

JAVACLASS (?:[a-zA-Z0-9-]+.)+[A-Za-z0-9$]+

As per the above JAVACLASS requires atleast a period (.) symbol to appear in the text. However in your logs it is just STDIO.

Replace your grok match by the following:-

match => { "message" => "^%{TIMESTAMP_ISO8601:JigsawTimestamp}%{SPACE}%{LOGLEVEL:JigsawLoglevel}%{SPACE}%{SPACE}%{HOSTNAME:ThreadName}%{SPACE}%{UUID:GUID}%{SPACE}%{WORD:JigsawClassName}%{SPACE}%{WORD:JigsawMethodName}%{SPACE}%{GREEDYDATA:JigsawLogMessage}" }

Also for easy understanding use output to redirect it to console by adding stdout plugin as shown below:-

output {
if "_grokparsefailure" not in [tags] {
elasticsearch { 
  hosts => ["localhost:9200"] 
}
stdout { codec => rubydebug }
}

It will make it easier for you to understand the error while processing data using Logstash.

Logstash to kibana multiline not working for custom message

1 Answers