Sort logs by timestamp in Kibana

Question

Timestamps don`t match I have and ELK (Elasticsearch, Logstash and Kibana) instance running and Filebeat sending logs from other machines, and I noticed that logs are shown in Kibana in different order that they arrived to the server (as you can see in the attached picture), for example, this is what is shown in Kibana, the second column is kibana timestamp and the third column is the server timestamp:

February 9th 2017, 11:53:11.714 11:53:04,904
February 9th 2017, 11:53:11.714 11:53:05,579
February 9th 2017, 11:53:11.714 11:53:05,581
February 9th 2017, 11:53:11.714 11:53:05,591
February 9th 2017, 11:53:11.714 11:53:04,441
February 9th 2017, 11:53:11.714 11:53:05,589

What I see in the log file is:

11:53:04,441
11:53:04,904
11:53:05,579
11:53:05,581
11:53:05,589
11:53:05,591

Is there any option to see the logs in Kibana in the same order that they are shown in the server? I was looking for it but I didn`t see any question which treat this topic, but I saw that it could be changing the Logstash configuration file 10-syslog-filter.conf, this is mine:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

Thanks in advance.

Kulasangar Kulasangar · Accepted Answer · 2017-02-09T10:14:28

I think that's because you haven't given the timestamp on which the events should be shown in Kibana, when you're creating the index in Kibana. You can do something like this:

mutate {
    add_field => { "received_time" => "%{syslog_timestamp}" }
    remove_field => ["syslog_timestamp"] <-- since you'll be using the recieved_time field here after
}
date {
    match => [ "received_time", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
    target => "received_time"
    locale => "en"
    timezone => "UTC"
}

The above is just a sample, so that you could reproduce. Note that I'm assuming syslog_timestamp as your server time which you mentioned in your Q.

So when you're creating a new index in Kibana, you'll be asked to select the time-field name in order to filter the events. What Kibana does is, it'll show the events based on the browser's time unless you select it from the drop down.

So once you process the logstash conf you should be able to see the field received_time which we created in the conf in the drop down. Thus you simply have to select it from the drop down , so that Kibana would show the events based on the timestamp you have selected. Also make sure that you don't have any timezone issues as well. If you're using UTC, please change the browser timezone as well from Advanced Settings > dateFormat:tz in Kibana. Hope it helps!

Sort logs by timestamp in Kibana

1 Answers