Logstash code to upload CSV file content into Elasticsearch

Question

I have CSV file and I want to import it into Elasticsearch 5.0.0 using Logstash.

This are first 2 lines of the CSV file:

Id,CameraId,ZoneId,Latitude,Longitude,Number,OriginalNumber,Country,OriginalCountry,CapturedOn,Status,Direction,Speed,Confidence,AvgDigitsHeight,MultiplateRate,ProcessingTimeOCR,Signaled,OcrImageId,EnvImageIds,CapturerId,CapturerType,IsAlarm,AlarmListIds,ReplicationId,ImagesUploaded
111,24,1,42.8,3.5,XXDFR,XXDFR,DE,DE,2017-03-04 12:06:20.0,0,1,0,99,21.0,3,16.0193003809306,0,0,[],null,null,0,[],0,0

I run this Logstash script:

input {
    file {
        path => ["/usr/develop/test.csv"]
        type => "core2"
        start_position => "beginning"
    }
}

filter {
    csv {
        columns => [
"Id","CameraId","ZoneId","Latitude","Longitude,"Number","OriginalNumber","Country","OriginalCountry","CapturedOn","Status","Direction","Speed","Confidence","AvgDigitsHeight","MultiplateRate","ProcessingTimeOCR","Signaled","OcrImageId","EnvImageIds","CapturerId","CapturerType","IsAlarm","AlarmListIds","ReplicationId","ImagesUploaded"
        ]
    }
}

output {
    stdout { codec => rubydebug }
    elasticsearch {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "mytest"
        document_type => "doc"
        document_id => "%{Id}"
        workers => 1
    }
}

I get this error:

logstash.agent ] fetched an invalid config {:config=>"input {\nfile {\npath => [\"/usr/develop/test.csv\"]\ntype => \"core2\"\nstart_position => \"beginning\" \n}\n}\nfilter {\ncsv {\nseparator => \",\"\ncolumns => [\"Id\",\"CameraId\",\"ZoneId\",\"Latitude\",\"Longitude,\"Number\",\"OriginalNumber\",\"Country\",\"OriginalCountry\",\"CapturedOn\"]\n}\n}\noutput {\nelasticsearch {\naction => \"index\"\nhosts => [\"localhost:9200\"]\nindex => \"test\"\ndocument_type => \"doc\"\ndocument_id => \"%{Id}\"\nworkers => 1\n}\nstdout { codec => rubydebug}\n}\n\n", :reason=>"Expected one of #, {, ,, ] at line 11, column 61 (byte 225) after filter {\ncsv {\nseparator => \",\"\ncolumns => [\"Id\",\"CameraId\",\"ZoneId\",\"Latitude\",\"Longitude,\""}

You're missing a double quote after Longitude: "Longitude, <-- here — Val

xProgramery xProgramery · Accepted Answer · 2017-07-21T01:42:02

Not sure if you caught this yet, but it's because you are missing a " for the column name "Longitude"

Logstash code to upload CSV file content into Elasticsearch

1 Answers