Logstash output works but fail to feed elasticsearch

Question

I am trying to pass my API custom log files to Kibana. For that I use myLogFiles.log -> Logstash -> Elasticsearch -> Kibana.

PROBLEM

Logstash output tells that everything is fine, but Elasticsearch remain empty (I checked that with kibana or elasticsearch web visualisation plugins)

QUESTION

How can I feed elasticsearch to visualise my data in kibana ?

DEBUG SO FAR

To have some debug I told Logstash to ouput in a .log file and on console. see my logstash.conf:

input {
    file {
        path => '/home/***/dev_logstach/acci.log'
    start_position => beginning
    }
}


filter {
    grok {
        match => {"message" =>  "%{WORD:key} %{WORD:userID} %{WORD:lakeID} %{WORD:ballID} %{NUMBER:longitude} %{NUMBER:latitude} %{TIMESTAMP_ISO8601:date_evenement}"}
    }
}


output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "testACCI"
  }
  stdout { codec => rubydebug }
  file {
   path => "/home/***/dev_logstach/testLog.log"
   create_if_deleted => true
  }
}

And a logstash output

{
            "userID" => "HJcOX",
        "@timestamp" => 2018-04-10T13:27:47.723Z,
          "latitude" => "-123.8",
           "message" => "{message:ACWWCI HJcOX qHYFM ABCCCC -22.5 -123.8 2018-04-10T09:11:06.173Z,level:info}",
              "path" => "/home/***/dev_logstach/acci.log",
    "date_evenement" => "2018-04-10T09:11:06.173Z",
          "@version" => "1",
            "lakeID" => "qHYFM",
               "key" => "ACWWCI",
         "ballID" => "ABCCCC",
              "host" => "sd-10****",
         "longitude" => "-22.5"
}

When I run again it to not log previous data, and when logstash run if I

 echo -e "new data line" >> acci.log

the new data appear in the logs. I then suppose that the data are send somewhere but I have no clue where and how can I move forward.

In elasticsearch log I just see a warning:

2018-04-10T17:01:52,507][WARN ][o.e.d.i.m.MapperService  ] [_default_] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type

P.S. I read some problem like that but it was due to date format mine is valide (YYYY-MM-DD...)

Evaldas Buinauskas Evaldas Buinauskas · Accepted Answer · 2018-04-11T09:53:00

Couple of options:

Create your elasticsearch index in advance
Create elasticsearch index template

This should resolve your issue.

Logstash output works but fail to feed elasticsearch

1 Answers