How to convert a string to data field in logstash filter plugin

Question

I'm new to ELK stack. exploring logstash and elastic search for the last one week. I'm finding difficulties to convert a time stamp(default considered as string type) to date type field. This the string "Thu May 18 06:39:44 CEST 2017" represents timestamp in tool log wanted to convert this string to date field using logstash filter Date plugin. I don't know how to make it. Any suggestion or help would be appreciated. Thanks !

filter {    
 csv {     
    columns =>["Make","Color","Price","Sold"]     
    separator =>","       
 }       
date
{      
match => ["Sold", "DD-MM-YYYY"] #  Not sure how to match timestamp in log file to convert to date type.        
target => "Sold"      
}      
mutate{      
   convert => ["Price","float"]    
   }      
 }

//Let's assume My csv file contains the below logs. the last field for timestamp iPhone,silver,260,Thu Sep 07 11:05:07 CEST 2017

iPhone,gold,400,Thu Oct 07 12:05:07 CEST 2017

@HatimStovewala . Thanks for your reply. pls find the config file. — Arockiyaraj
EEE MMM dd HH:mm:ss `CEST` yyyy I haven't tested it. Note that the EEE and MMM are locale-dependent. — baudsp

Hatim Stovewala Hatim Stovewala · Accepted Answer · 2017-10-18T12:54:51

Set up your Index first by below command. (... replace it with your rest of the columns and their type. Refer Elasticsearch Put Mapping)

PUT date
{
  "mappings": {
    "date": {
      "properties": {
        ...,
        "TimeStamp": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss zzz yyyy"
        },
        ...
      }
    }
  }
}

Now, you can go ahead with logstash to push data into ES. Remove the date filter from logstash config file. ES will parse the string into date.

Note : While testing it with sample data, I realised that CEST is not accepted as a timezone. I guess you will need to replace CEST with CST. There is some issue with this kind of parsing, refer.

If the timezone is fixed to CEST in your data then you can do this work around:

PUT date
{
  "mappings": {
    "date": {
      "properties": {
        ...,
        "TimeStamp": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss 'CEST' yyyy"
        },
        ...
      }
    }
  }
}

How to convert a string to data field in logstash filter plugin

1 Answers