What to do to stop elasticsearch indexes being created automatically in the wrong year

Question

I noted this at the turn of the year, I have asked about it in #elasticsearch and #logstash a few times but never had a response that explains what best to do to stop it. I also found this post on the mailing list, but it doesn't discuss how to stop it from happening in the future.

Currently in elasticsearch I have a number of indexes which have been created automatically for November 2015 - December 2015. The indexes have been created automatically when reading logs from logstash and sending the results to elasticsearch. The logs in question are ones where the year is not specified in the timestamp, I have then taken the timestamp and used the date match function to set that timestamp as the @timestamp system field. EG:

{"message":"Dec 31 12:03:00 server sudo:   zabbix : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/asterisk -rx meetme","@version":"1","@timestamp":"2015-12-31T12:03:00.000Z","file":"/var/log/secure","host":"server","offset":"74","type":"syslog","syslog_timestamp":"Dec 31 12:03:00","syslog_hostname":"server","syslog_program":"sudo","syslog_message":"  zabbix : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/asterisk -rx meetme","tags":["syslog"],"syslog_severity_code":5,"syslog_facility_code":1,"syslog_facility":"user-level","syslog_severity":"notice"}

The @timestamp value for this field is set to:

December 31st 2015, 12:03:00.000

I believe that these logs were processed by logstash after the year changed to 2015, I am not sure, but I may have restarted logstash before the log was processed. The reason I believe I restarted logstash, is that I believe the year is checked when logstash starts, if a date filter is applied and no year is specified in the event then Logstash assumes the year must be the current year taken when the system restarted and elasticsearch applies those events against the index in the future.

The ability to create the indexes automatically is great, and I don't want to stop doing that, but is there any way to stop Logstash/elasticsearch from automatically generating indexes at the end of the current year when it processes logs with event timestamps from the end of last year so that the logs will go in to the correct index (that is, last years indexes)

R. Arruda R. Arruda · Accepted Answer · 2015-04-28T12:27:05

According to the code in the date filter:

https://github.com/elastic/logstash/blob/v1.4.2/lib/logstash/filters/date.rb#L153

When logstash doesnt know which year the log event happened (cant parse the year), it will assume it is the current year. (queried at run time)

See the year in effect is just missing (which will be then assumed to be the current year), the only thing you could do is just drop events in a condition of time interval.

I think the solution you are looking for is this:

Its possible to match on the "future"* year/month, and drop these events:

 filter {
   mutate {
    add_field => { "year_month" => "%{+yyyyMM}" }
   }
   if [year_month] =~ /^20151[012]$/ {
     drop { }
   }
 }

*shows as in the future, but they actually belong to the previous year.

It's possible to do fancier regexps to match the future events a bit better.

What to do to stop elasticsearch indexes being created automatically in the wrong year

2 Answers