Logstash - Error log event date going as string to ES

Question

I'm using Logstash to forward error logs from app servers to ES. Everything is working fine except that log timestamp going as string to ES.

Here is my log format

[Date:2015-03-25 01:29:09,554] [ThreadId:4432] [HostName:AEPLWEB1] [Host:(null)] [ClientIP:(null)] [Browser:(null)] [UserAgent:(null)] [PhysicalPath:(null)] [Url:(null)] [QueryString:(null)] [Referrer:(null)] [Carwale.Notifications.ExceptionHandler] System.InvalidCastException: Unable to cast object of type 'Carwale.Entity.CMS.Articles.ArticleDetails' to type 'Carwale.Entity.CMS.Articles.ArticlePageDetails'. at Carwale.Cache.Core.MemcacheManager.GetFromCacheCore[T](String key, TimeSpan cacheDuration, Func`1 dbCallback, Boolean& isKeyFirstTimeCreated)

Filter configuration for logstash forwarder

filter {  

    multiline {
            pattern => "^\[Date:%{TIMESTAMP_ISO8601}"
            negate => true
            what => "previous"
        }   

    grok {
        match => [ "message", "(?:Date:%{TIMESTAMP_ISO8601:log_timestamp})\] \[(?:ThreadId:%{NUMBER:ThreadId})\] \[(?:HostName:%{WORD:HostName})\] \[(?:Host:\(%{WORD:Host})\)\] \[(?:ClientIP:\(%{WORD:ClientIP})\)\] \[(?:Browser:\(%{WORD:Browser})\)\] \[(?:UserAgent:\(%{WORD:UserAgent})\)\] \[(?:PhysicalPath:\(%{WORD:PhysicalPath})\)\] \[(?:Url:\(%{WORD:Url})\)\] \[(?:QueryString:\(%{WORD:QueryString})\)\] \[(?:Referrer:\(%{WORD:Referrer})\)\] \[%{DATA:Logger}\] %{GREEDYDATA:err_message}" ]  
    }

    date {
        match => [ "log_timestamp", "MMM dd YYY HH:mm:ss","MMM  d YYY HH:mm:ss", "ISO8601" ]
        target => "log_timestamp"
    }

    mutate {
        convert => ["ThreadId", "integer"]
    }
}

How I can make it date in ES? Please help. Thanks in advance.

I would guess that it doesn't like to overwrite fields. Use another field name. I would suggest using the default (@timestamp). — Alain Collins
Have you looked at creating a mapping template for your logstash indexes? — AaronM

VinothNair VinothNair · Accepted Answer · 2015-07-22T12:15:57

I had the similar issue. Now fixed it with the below workaround.

      grok {
        match => {
            "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}[T ]%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}" 

             }
        }

  grok{
    match => {
        "second" => "(?<asecond>(^[^,]*))"  }
    }

mutate {
        add_field => { 
        "timestamp" => "%{year}-%{month}-%{day} %{hour}:%{minute}:%{asecond}"
                             }
    }

    date{  match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ] timezone=> "UTC" target => "log_timestamp" }

Thanks,

Logstash - Error log event date going as string to ES

1 Answers