Why ElasticSearch sometimes doesn't show latest log that Logstash sent?

Question

My Logstash sends log to ElasticSearch, in Logstash output file I see that new logs are sent but they are not shown when click refresh in Kibana Discover page. The Discover page shows logs ~1 hours ago.

What could cause this issue? Is there any way to check if a record is in ElasticSearch rather than using Kibana?

The timestamp in my log file is like this: [2020-09-02 13:53:07,392Z]

Here is the pipeline.yml my logstash uses:

input {
    #stdin {}
    beats {
        port => "5055"
    }
    # file {
}
filter {
    grok {
        patterns_dir => "C:\logstash-7.4.2\patterns"
        match => { "message" => "^\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{LOGLEVEL:level}\]\[%{Thread:thread}\]\[%{JAVACLASS:class}\](\[\d*\])? %{GREEDYDATA:msg}" }
   }
   if "_grokparsefailure" in [tags] {
       drop{}
   }
}
output {
    file {
        path => "C:\logstash-7.4.2\logstash_output.txt"
    }
    elasticsearch {
        hosts => [ "localhost:9200" ]
        index => "ts_services-%{+YYYY.MM.dd}"
    }
}

I see the latest logs after I deleted the existing index pattern and created a new one. But I don't think this is solution.

That's probably a timestamp issue. Kibana interprets the timestamp in the browser timezone so if your logs don't have a timezone at the end, they might not show correctly. Can you show how the timestamps in your log file look like as well as your Logstash configuration? — Val
It can also be an issue with Logstash naming convention, versus the way aliases are used in you logstash/Kibana. Could you also add you index settings ? — Jaycreation
@Val I added timestamp format and Logstash config to my question. — user1532146
@Jaycreation index setting in Logstash yml file is now in my question. — user1532146
Just found that the default index pattern defined in Kibana has month as 08: someservice-2020.08*. After "08" is removed the latest log is shown in Kibana. Thanks @Jaycreation. — user1532146

Jaycreation Jaycreation · Accepted Answer · 2020-09-03T06:23:57

To follow the discussion in the comments. The issue here is a very common one.

The index pattern in Kibana was to specific:

ts_services-2020-08*

So data in September where ignored by Kibana, even if they are in Elasticsearch.

Changing the Kibana index pattern will fix the issue:

ts_services*

But you will have to correct all custom visualizations.

Why ElasticSearch sometimes doesn't show latest log that Logstash sent?

1 Answers