Logstash parsing csv date

Question

I have a problem parsing date from csv and I cannot find the problem with (one would assume) simple date - dd/MM/yy. Here's structure of my csv file:

Date,Key-values,Line Item,Creative,Ad unit,Creative size,Ad server impressions,Ad server clicks,Ad server CTR
04/04/16,prid=DUBAP,Hilton_PostAuth 1,Stop Clicking Around - 300x250,383UKHilton_300x250,300 x 250,31,0,0.00%
04/04/16,prid=DUBAP,Hilton_PostAuth 2,16-0006_Auction_Banners_300x250_cat4,383UKHilton_300x250,300 x 250,59,0,0.00%

and my logstash.config file:

input {
  file {
    path => "/Users/User/*.csv"
    type => "core2"
    start_position => "beginning"    
  }
}
filter {
  csv {
      columns => ["Date","Key-values","Line Item","Creative","Ad unit","Creative size","Ad server impressions","Ad server clicks","Ad server CTR"]
      separator => ","
  }
  date {
    match => ["Date", "dd/MM/YY"]
  }
  mutate {convert => ["Ad server impressions", "float"]}
  mutate {convert => ["Ad server clicks", "float"]}
  mutate {convert => ["Ad server CTR", "float"]}
}
output {  
    elasticsearch {
        action => "index"
        hosts => "localhost"
        index => "test1"
        workers => 1
    }
    stdout {}
}

I have also tried combinations with date being "dd/MM/yy" with no luck, Date is not indexed as date, and I can select only @timestamp in Kibana..

I think there must be a simple thing I'm just missing but as for this moment I cannot find it..

Cheers!

EDIT 1:

Please find my console output when logstash starts and how data is being processed:

Settings: Default pipeline workers: 4
Pipeline main started
Failed parsing date from field {:field=>"Date", :value=>"Date", :exception=>"Invalid format: \"Date\"", :config_parsers=>"dd/MM/YY", :config_locale=>"default=en_US", :level=>:warn}
2016-05-06T20:32:48.034Z Pawels-MacBook-Air.local Date,Key-values,Line Item,Creative,Ad unit,Creative size,Ad server impressions,Ad server clicks,Ad server CTR
2016-04-03T23:00:00.000Z Pawels-MacBook-Air.local 04/04/16,prid=DUBAP,Hilton_PostAuth 1,Stop Clicking Around - 300x250,383UKHilton_300x250,300 x 250,31,0,0.00%

It still loads it into Elasticsearch but in Kibana there's no 'Date' field - I can only use @timestamp

Cheers

alpert alpert · Accepted Answer · 2016-05-06T21:47:55

Actually what date filter does is:

The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.

So with that configuration it reads your date and use it as timestamp field. If you want to use it as a seperate field, configure as:

date {
    match => ["Date", "dd/MM/yy"]
    target => "Date"
}

Logstash parsing csv date

1 Answers