Parse Multiline JSON with Arrays using grok in logstash

Question

Hi this is my first time using Kibana. I am trying to parse the below input file to logstash and put it to elastic search for use in Kibana.

{
  "ASRtest": {
    "ASRHDR": "This is asr HDR",
    "ASRTestType": "DevTest",
    "Scenario": [
      {
        "ScenarioNumber": 1,
        "ScenarioName": "HTTP Validation",
        "ScenarioDescription": "Validate if the API alows access over HTTP",
        "ScExecutionStatus": "Execution Complete",
        "ScenarioStatus": "In-Complete",
        "ScenarioSeverity": false,
        "TestCase": [
          {
            "TestCaseNumber": 1,
            "TestCaseName": "HTTP Validation - using POST method ",
            "TcExecutionStatus": "Execution Error",
            "TcStatus": "NA",
            "TcSeverity": "NA"
          }
        ]
      },
      {
        "ScenarioNumber": 2,
        "ScenarioName": "Server Platform/Version Disclosure",
        "ScenarioDescription": "Validate if API disclose server information",
        "ScExecutionStatus": "Execution Complete",
        "ScenarioStatus": "Failure",
        "ScenarioSeverity": "Medium",
        "TestCase": [
          {
            "TestCaseNumber": 1,
            "TestCaseName": "Server Platform/Version Disclosure - using POST method ",
            "TcExecutionStatus": "Executed Successfully",
            "TcStatus": "Failure",
            "TcSeverity": "Medium"
          }
        ]
      }
    ]
  }
}

I want all the fields in the input to be available for the dashboard charts. Could anyone explain how to parse this multiline JSON file to logstash ==> elasticsearch. I tried a sample config file but I wa not able to produce the desired output.

Kulasangar Kulasangar · Accepted Answer · 2017-01-15T17:03:12

You may have to use multiline codec which could ideally allow joining of multiline messages from files into a single event. The input could look something like this:

input 
{   
    file 
    {
        codec => multiline
        {
            pattern => '^\{'
            negate => true
            what => previous                
        }
        path => ["path to your json file/.json"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
        exclude => "*.gz"
    }
}

This thread and SO could be useful. Hope it helps!

Parse Multiline JSON with Arrays using grok in logstash

1 Answers