how to override timestamp field coming from json in logstash

Question

I have the following below kind of json present with me to be dumped in elastic search using filebeat {"@timestamp":"2017-02-10T06:30:51.424Z","beat":{"hostname":"myhostname","name":"mydevice-logger","version":"5.2.0"},"fields":{"device_type":"mydevice","env":"staging"},"metricset":{"module":"system","name":"cpu","rtt":211},"system":{"cpu":{"cores":4,"idle":{"pct":0.000000},"iowait":{"pct":0.000000},"irq":{"pct":0.000000},"nice":{"pct":0.000000},"softirq":{"pct":0.000000},"steal":{"pct":0.000000},"system":{"pct":0.000000},"user":{"pct":0.000000}}},"tags":["automata","box"],"type":"metricbeat-test-log"}

my logstash( version 5.1.1) config contains, input, filter and output like below -

input { 
  beats {
        port => 5046
        codec => json
  }
}

filter {
    if ...{}
    else if [type] == "metricbeat-test-log" {

      date {
        match => ["@timestamp", "ISO8601"]
      }

      }
    }

}


output {
    if ...{}
    else if [type] == "metricbeat-test-log" {
        stdout { codec => rubydebug   }
    }
}

The type is right however the date filter is not working . The @timestamp finally takes current timestamp always . I want to replace it with original @timestamp present in json.

Are you really using Filebeat to write Metricbeat's JSON data to Elasticsearch? — A J
Yes the reason is its an offline device. I collect metricbeat log and putting them in to elastic search — Ankit Kulkarni
though I can simply use a python script to dump the whole json into ES however I find using filebeat more practical for my case — Ankit Kulkarni

Bren Bren · Accepted Answer · 2017-02-14T16:04:09

You should use the target setting in the date filter :

From https://www.elastic.co/guide/en/logstash/5.1/plugins-filters-date.html#plugins-filters-date-target

Target

Value type is string

Default value is "@timestamp"

Store the matching timestamp into the given target field. If not provided, default to updating the @timestamp field of the event.

how to override timestamp field coming from json in logstash

2 Answers