Logstash - parses only one JSON event

Question

I am using ELK 5.3.0. I am trying to parse simple JSON document. It does work creating key/values, however it writes in Elasticsearch only one event. And it does that randomly. Sometimes is first, sometimes second or third. But is is always one event.

Filesetup (Created in Mac. One line per JSON object), three events:

{"timestamp":"2012-01-01 02:00:01", "severity":"ERROR", "messages":"Foo failed", "fieldone": "I am first entry... if the value of a field one", "fieldtwo": "ttthis if the value of a field two"} {"timestamp":"2013-01-01 02:04:02", "severity":"INFO", "messages":"Bar was successful", "fieldone": "I am second entry... if the value of a field one", "fieldtwo": "this if the value of a field two"} {"timestamp":"2017-01-01 02:10:12", "severity":"DEBUG", "messages":"Baz was notified", "fieldone": "I am third entry... if the value of a field one", "fieldtwo": "this if the value of a field two"}

Filebeatsetup:

- input_type: log
  paths: Downloads/elk/small/jsontest.log
  document_type: jsonindex

Logstashsetup:

filter {
  if [@metadata][type] == "jsonindex" {
        json {
            source => "message"
        }            
    }
  }

Logstash output (shows three events):

{
"severity" => "DEBUG",
"offset" => 544,
"@uuid" => "a316bb67-98e5-4551-8243-f8538023cfd9",
"input_type" => "log",
"source" => "/Users/xxx/Downloads/elk/small/jsontest.log",
"fieldone" => "this if the value of a field one",
"type" => "jsonindex",
"tags" => [
[0] "beats_input_codec_json_applied",
[1] "_dateparsefailure"
],
"fieldtwo" => "this if the value of a field two",
"@timestamp" => 2017-05-08T11:25:41.586Z,
"@version" => "1",
"beat" => {
"hostname" => "C700893",
"name" => "C700893",
"version" => "5.3.0"
},
"host" => "C700893",
"fingerprint" => "bcb57f445084cc0e474366bf892f6b4ab9162a4e",
"messages" => "Baz was notified",
"timestamp" => "2017-01-01 02:10:12"
}
{
"severity" => "INFO",
"offset" => 361,
"@uuid" => "6d4b4401-a440-4894-b0de-84c97fc4eaf5",
"input_type" => "log",
"source" => "/Users/xxx/Downloads/elk/small/jsontest.log",
"fieldone" => "this if the value of a field one",
"type" => "jsonindex",
"tags" => [
[0] "beats_input_codec_json_applied",
[1] "_dateparsefailure"
],
"fieldtwo" => "this if the value of a field two",
"@timestamp" => 2017-05-08T11:25:41.586Z,
"@version" => "1",
"beat" => {
"hostname" => "C700893",
"name" => "C700893",
"version" => "5.3.0"
},
"host" => "C700893",
"fingerprint" => "bcb57f445084cc0e474366bf892f6b4ab9162a4e",
"messages" => "Bar was successful",
"timestamp" => "2013-01-01 02:04:02"
}
{
"severity" => "ERROR",
"offset" => 177,
"@uuid" => "d9bd0a0b-0021-48fd-8d9e-d6f82cd1e506",
"input_type" => "log",
"source" => "/Users/xxx/Downloads/elk/small/jsontest.log",
"fieldone" => "this if the value of a field one",
"type" => "jsonindex",
"tags" => [
[0] "beats_input_codec_json_applied",
[1] "_dateparsefailure"
],
"fieldtwo" => "this if the value of a field two",
"@timestamp" => 2017-05-08T11:25:41.586Z,
"@version" => "1",
"beat" => {
"hostname" => "C700893",
"name" => "C700893",
"version" => "5.3.0"
},
"host" => "C700893",
"fingerprint" => "bcb57f445084cc0e474366bf892f6b4ab9162a4e",
"messages" => "Foo failed",
"timestamp" => "2012-01-01 02:00:01"
}

ElasticSearch (document viewed in as JSON):

"tags": [
      "beats_input_codec_json_applied",
      "_dateparsefailure"
    ],

There is no JSON failure. _dateparsefailure is expected.

What is going on in here?

EDIT (Solution): After some time, I figured I was shooting myself in the leg. Since I am parsing many different logs and also log types, I need to make certain I do not have duplicates, this in my Logstash output section I have this piece of code to ensure no duplicate log entires:

uuid {
        target => "@uuid"
        overwrite => true
    }

    fingerprint {
        source => ["message"]
        target => "fingerprint"
        key => "78787878"
        method => "SHA1"
        concatenate_sources => true
    }
}

End also in the same section I call ElasticSearch like this:

if [@metadata][type] == "jsonindex" {
        elasticsearch {
            hosts => [ "localhost:9200" ]
            index => "%{[@metadata][type]}"
            document_id => "%{fingerprint}"
        }
    }

Since my JSON objects do not contain message property, it is always virtually identical:

fingerprint {
            source => ["message"]

...

Small edit to index creation fixed the problem:

if [@metadata][type] == "jsonindex" {
            elasticsearch {
                hosts => [ "localhost:9200" ]
                index => "%{[@metadata][type]}"                
            }
        }

Joseph M Tsai Joseph M Tsai · Accepted Answer · 2017-05-08T15:17:21

your data need to separate by lines

the grok need to parse it as different line,it will become 3 data

for example :

{"timestamp":"2012-01-01 02:00:01", "severity":"ERROR", "messages":"Foo failed", "fieldone": "I am first entry... if the value of a field one", "fieldtwo": "ttthis if the value of a field two"} 
{"timestamp":"2013-01-01 02:04:02", "severity":"INFO", "messages":"Bar was successful", "fieldone": "I am second entry... if the value of a field one", "fieldtwo": "this if the value of a field two"} 
{"timestamp":"2017-01-01 02:10:12", "severity":"DEBUG", "messages":"Baz was notified", "fieldone": "I am third entry... if the value of a field one", "fieldtwo": "this if the value of a field two"}

you are in one line so the result is parse the last one which means the timestamp is the last one

"timestamp":"2017-01-01 02:10:12

if you change by line ,i think maybe not but you could use this

- input_type: log
  paths: Downloads/elk/small/jsontest.log
  document_type: jsonindex
  multiline.pattern: '^{"timestamp":"[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}, '
  multiline.negate: true
  multiline.match: after

add the mutiline to change ,but i am afraid your data not separate by lines

Logstash - parses only one JSON event

1 Answers