Kibana shows logs in discover with Delay

Question

I have a setup with Elastic Search, Log Stash, Kibana and Filebeat with versions as shown below.

1) Elastic Search: 6.2.4 2) LogStash: 6.2.4 3) kibana: 6.2.4

My Logstash pipeline is as below:

sudo vim /etc/logstash/conf.d/02-beats-input.conf

input {
    beats {
        port => "5044"
        ssl => true
        ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
        ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    }
}

filter {
  #If log line contains tab character followed by 'at' then we will tag that 
  entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }

  #Grokking Spring Boot's default log format
  grok {
    match => { "message" => "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} % 
      {TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- \[(?<thread>[A-Za-z0- 
      9-]+)\] [A-Za-z0-9.]*\.(?<class>[A-Za-z0-9#_]+)\s*:\s+(? 
      <logmessage>.*)"}
    }

  #grok {
    #  match => { "message" => "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} 
    # %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(? 
    # <logmessage>.*)"}
  #}

  grok {
    match => { "source" => "/var/log/containers/%{DATA:pod_name}_% 
    {DATA:namespace}_%{GREEDYDATA:container_name}-%{DATA:container_id}.log" 
  }
  remove_field => ["source"]
}

sudo vim /etc/logstash/conf.d/30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Elastic search, Kibana, Logstash are running in one VM where as application and Filebeat are running on a different VM.

Currently there is a discrepancy with date between these two VMs, which I need to fix yet.

The latest log on kibana discover is as below with a specific time stamp.

message:
  {"log":"TACACS+: No port assigned for host, \"XX.XX.XX.XX\". Using default 
  port 49 instead.\n","stream":"stdout","time":"**2018-05- 
  17T00:58:09.401752809Z**"}
  @timestamp:
  May 16th 2018, 17:58:09.408

The latest log at the application as below with a specific time stamp.

{"log":"TACACS+: No port assigned for host, \"XX.XX.XX.XX\".  Using default 
port 49 instead.\n","stream":"stdout","time":"**2018-05- 
17T06:06:44.365607578Z**"}

If you see the above two logs, it is clear that kibana is showing up the logs with some delay, particularly in the above case it has delay of around 5 hours. I also see that the delay is keep getting incremented. I see all the logs on the kibana though. Issue is the delay i am seeing.

Could some one help me understand this behavior ? Is this because the time discrepancy between the two VMs ? both are at PDT time zone. The logs size should be small enough and I dont expect any throttling kicks in.

Please let me know if you need any other details on this.

did you find any solution to your problem? I'm seeing the same issue with my ELK stack as well — Mohsin Kamal
I have the same problem. The thing is I can get the list of documents by execute the query: GET filebeat-2020-04-29/_search { "query": { "match_all": {} } } — duybinh0208

duybinh0208 duybinh0208 · Accepted Answer · 2020-04-29T11:09:30

I believe the mismatch in time/timezone caused the issue. Just sync it and the problem will be solved.

Kibana shows logs in discover with Delay

1 Answers