Logstash parse JSON into individual events

Question

Im trying an HTTP Poller which returns me the response in the below format (this is a single line JSON) .

{"total":3,"offset":1,"len":50,"workflows":[
{"appName":"test1","createdTime":"Wed, 11 May 2016 13:30:28  GMT","startTime":"Wed, 11 May 2016 13:30:28 GMT","endTime":"Wed, 11 May 2016 13:31:06 GMT","status":"SUCCEEDED"},
{"appName":"test2","createdTime":"Wed, 11 May 2016 13:30:28 GMT","startTime":"Wed, 11 May 2016 13:30:28 GMT","endTime":"Wed, 11 May 2016 13:31:06 GMT","status":"SUCCEEDED"},
{"appName":"test3","createdTime":"Wed, 11 May 2016 13:30:28 GMT","startTime":"Wed, 11 May 2016 13:30:28 GMT","endTime":"Wed, 11 May 2016 13:31:06 GMT","status":"SUCCEEDED"}
]
}

The requirement for me is to store each of the workflow item (array element) as a separate event in the elastic search. To be specific , I want to extract the appName, createdTime, Status for each record and pass this individual event to the ElasticSearch output plugin.

Can you help on this?

The logstash conf file is as below

input {
  http_poller 
  {
    urls => 
    {
      mycall => 
      {
        method => "GET"
        url => "http://myip/url"            
      }
    }
    tags => 'data'
    request_timeout =>60
    interval => 1200
    codec => "json"
    metadata_target => "http_poller_metadata"
  }


  }

output {    
stdout  
{   

 codec => rubydebug }
}

alpert alpert · Accepted Answer · 2016-05-19T16:21:36

With split filter you can split and with mutate you can extract the fields:

Conf:

split {
    field => "workflows"
    terminator => ","
    }
mutate {
   rename => {
    "[workflows][appName]" => "appName"
    "[workflows][createdTime]" => "createdTime"
    "[workflows][startTime]" => "startTime"
    "[workflows][endTime]" => "endTime"
    "[workflows][status]" => "status"
   }
   remove_field => ["workflows", "total", "offset", "len"]
}

Result:

{
       "@version" => "1",
     "@timestamp" => "2016-05-19T16:35:50.177Z",
           "host" => "Alpers-MacBook-Pro.local",
        "appName" => "test1",
    "createdTime" => "Wed, 11 May 2016 13:30:28  GMT",
      "startTime" => "Wed, 11 May 2016 13:30:28 GMT",
        "endTime" => "Wed, 11 May 2016 13:31:06 GMT",
         "status" => "SUCCEEDED"
}
{
       "@version" => "1",
     "@timestamp" => "2016-05-19T16:35:50.177Z",
           "host" => "Alpers-MacBook-Pro.local",
        "appName" => "test2",
    "createdTime" => "Wed, 11 May 2016 13:30:28 GMT",
      "startTime" => "Wed, 11 May 2016 13:30:28 GMT",
        "endTime" => "Wed, 11 May 2016 13:31:06 GMT",
         "status" => "SUCCEEDED"
}
{
       "@version" => "1",
     "@timestamp" => "2016-05-19T16:35:50.177Z",
           "host" => "Alpers-MacBook-Pro.local",
        "appName" => "test3",
    "createdTime" => "Wed, 11 May 2016 13:30:28 GMT",
      "startTime" => "Wed, 11 May 2016 13:30:28 GMT",
        "endTime" => "Wed, 11 May 2016 13:31:06 GMT",
         "status" => "SUCCEEDED"
}

Logstash parse JSON into individual events

1 Answers