import json file data into elastic search using logstash

Question

i want to import json file data into elastic search. here is my config file of logstash--

input { file { type => "json" path => "C:\Users\Desktop\newJSON.json" start_position => "beginning" sincedb_path => "\dev\null" } }

output { stdout { codec => rubydebug } elasticsearch { hosts => "localhost:9200" index => "jsondata1" } }

And here is my json file---

{ "fruit": "Apple", "size": "small", "color": "Red" }, { "fruit": "Papaya", "size": "Large", "color": "Yellow" "test" : "sweet" }

i executed above config file using this command----

logstash -f logstashcon.conf

but i got data like below in elastic search index--

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 10,
    "max_score": 1,
    "hits": [
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablA",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.302Z",
          "@version": "1",
          "host": "user-102",
          "message": "{\r",
          "type": "json"
        }
      },
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablB",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.694Z",
          "@version": "1",
          "host": "user-102",
          "message": "    \"fruit\": \"Apple\",\r",
          "type": "json"
        }
      },
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablE",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.696Z",
          "@version": "1",
          "host": "user-102",
          "message": "},\r",
          "type": "json"
        }
      },
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablC",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.695Z",
          "@version": "1",
          "host": "user-102",
          "message": "    \"size\": \"Large\",\r",
          "type": "json"
        }
      },
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablD",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.696Z",
          "@version": "1",
          "host": "user-102",
          "message": "    \"color\": \"Red\"\r",
          "type": "json"
        }
      },
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablG",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.698Z",
          "@version": "1",
          "host": "user-102",
          "message": "\"fruit\": \"Papaya\",\r",
          "type": "json"
        }
      },
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablJ",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.699Z",
          "@version": "1",
          "host": "user-102",
          "message": "}\r",
          "type": "json"
        }
      },
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablH",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.699Z",
          "@version": "1",
          "host": "user-102",
          "message": "    \"size\": \"Large\",\r",
          "type": "json"
        }
      },
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablF",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.698Z",
          "@version": "1",
          "host": "user-102",
          "message": "{\r",
          "type": "json"
        }
      },
      {
        "_index": "jsondata1",
        "_type": "json",
        "_id": "AWNniXbgMkzPgBTTablI",
        "_score": 1,
        "_source": {
          "path": "C:\\Users\\Desktop\\newJSON.json",
          "@timestamp": "2018-05-16T06:00:48.699Z",
          "@version": "1",
          "host": "user-102",
          "message": "    \"color\": \"Yellow\"\r",
          "type": "json"
        }
      }
    ]
  }
}

Please help me to get correct output Thank you!

Answer 1

You need to use the JSON filter plugin for logstash to structure your output,

For example, if you have JSON data in the message field, your filter will be,

filter {
  json {
    source => "message"
  }
}

This is what the documentation says,

It takes an existing field which contains JSON and expands it into an actual data structure within the Logstash event.

Please read more about usage and example here.