How can I sync up log times in logstash or kibana

0
votes

I have two logs from different sources that I am pulling into elasticsearch via logstash and then visualizing with kibana. My problem is that the start time of each log should be the same however, one of the logs is recording entries incorrectly 30 minutes ahead.

How do I correct this time issue, is there a logstash filter that can subtract 30 minutes from the time or can I add some advanced query in Kibana?

I prefer to update the log entries in logstash.

1
logstashkibanaelastic-stacklogstash-configuration
Where does the dates come from? Parsed from the file or just the @timestamp added by logstash. Are you using one logstash or two? If two, are they on the same machine? – baudsp
@baudsp the dates are coming from the logfiles themselves. I'm just running one logstash over a couple of hours of each log file like a batch job, so I'm not running logstash continuously. – binarylegit
I've tried to find a way to change the value @timestamp field with the ruby filter but did not succeed, so I tried to see if the problem was coming from your setup. – baudsp

1 Answers

1
votes

Ideally, you may want to fix your logs so that the correct times are listed in the file. If your problem is in the data, it's easier to fix it there instead of wrestling with extra rules downstream.

But, you can manipulate the Logstash timestamp, if you need to.

The @timestamp field on each event has a lot in common with Ruby's Time class. In particular, it implements the same + and - operators which allow you to add or subtract some number of seconds:

filter {
    ruby {
        code => "
                event['@timestamp'] = LogStash::Timestamp.new(event['@timestamp'] + (30 * 60))
        "
    }
}