Is the @timestamp field needed when using Logstash to store in Elasticsearch?

4
votes

I have the following setup: I have a Java tool which sends JSON messages to RabbitMQ. They look like this:

{
"a": 0,
"b": 1,
"c": 2
}

Now I use Logstash to read the RabbitMQ queue and store them into Elasticsearch, so I can analyze the data with Kibana. The JSON stored in Elasticsearch looks like this:

{
"a": 0,
"b": 1,
"c": 2,
"@version": "1",
"@timestamp": "2014-01-22T19:05:19.136Z"
}

I don't think the @timestamp field will be of any use for what I'm doing. When I use cURL to store the same JSON in Elasticsearch, only the @version field is there, the @timestamp field is not present. Is there any way to configure Logstash to not save @timestamp?

2
javajsonelasticsearchlogstashkibana
Torben; would you be willing to share your knowledge on how you get logstash to read from the rabbitMQ server and also how do you remove the messages ElasticSearch reads from logstash? Does all this data just accumulate on your server? – vbNewbie

2 Answers

2
votes

When a message is read by Logstash, Logstash treat the message as a Event. An event will have a timestamp and message log. Thus, the @timestamp field is requisite.

Therefore, if you want to delete the @timestamp field, it will causes an error. Logstash can't output the event to the elasticsearch. 

Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `tv_sec' for nil:NilClass
    at RUBY.sprintf(file:/tmp/logstash-1.2.1-flatjar.jar!/logstash/event.rb:239)
    at org.jruby.RubyString.gsub(org/jruby/RubyString.java:3062)
    at RUBY.sprintf(file:/tmp/logstash-1.2.1-flatjar.jar!/logstash/event.rb:225)
    at RUBY.receive(file:/tmp/logstash-1.2.1-flatjar.jar!/logstash/outputs/elasticsearch.rb:153)

So far, not all @-prefix fields causes error, only remove @timestamp will cause this error.

1
votes

Any @-prefixed field is used internally by Logstash. Removing them tends to cause errors.

For example, I tried the following config file with Logstash 1.3.3:

input { 
    generator {
        type => "timestrip"
        message => "This is a test message."
        count => 1
    }
}

filter {
    mutate {
        remove_field => ["@timestamp"]
    }
}

output {
    elasticsearch_http {
        host => "127.0.0.1"
        flush_size => 1
    }
}

The ES output reports "Failed to flush outgoing items" with the following exception:

NoMethodError: undefined method `tv_sec' for nil:NilClass