Issue importing CSV file with Logstash

Question

I have a CSV file, and I want to import it in my Elastic Search.

I am on Windows 10 and I also have a Kibana be able to browse data once imported. I use Logstash to try to make this import.

All of my services (Kibana, ES and Logstash) are running on my localhost.

I tried with a the following Logstash configuration file (my csv file is in the correct path):

input {
  file {
    path => ["E:\Programmes\Logstash\logstash-2.2.0\data\*.csv"]
    start_position => "beginning"
  }
}

filter {
  csv {
    columns => ["billing_entity","invoice","company","username","reference","line_number","recipient_number","zone","network","date","time","country","duration","cost","currency","call_origin","billing_type"]
    separator => ";"
  }

  #grok {
  # match => { "call" => "%{WORD:billing_entity} %{WORD:invoice} %{WORD:company} %{WORD:username} %{WORD:reference} %{NUMBER:line_number} %{NUMBER:recipient_number} %{WORD:zone} %{WORD:network} %{DATE:date} %{TIME:time} %{WORD:country} %{WORD:duration} %{NUMBER:cost} %{WORD:currency} %{WORD:call_origin} %{WORD:billing_type}" }
  #}
}

output {
  elasticsearch {
    action => "index"
    index => "call_samples"
    #index => "call-%{+YYYY.MM.dd}"
    hosts => "localhost"
    workers => 1
  }
}

As you can see I tried to use 'csv' or 'grok' filter.

Then I launched in verbose mode logstash with this configuration file :

logstash.bat -f ..\conf\logstash.conf -v > logfile.txt

EDIT : after each try, I delete the generated sincedb files to simulate changes. But anyway I noticed they are empty

But in the logs I see nothing relevant :

message=>"Using mapping template from" message=>"Attempting to install template" message=>"New Elasticsearch output" message=>"Registering file input" message=>"No sincedb_path set,generating o.... message=>"Using mapping template from ... message=>"Attempting to install template" message=>"New Elasticsearch output" message=>"Starting pipeline" message=>"Pipeline started"

Quite alike my file is ignored .... I also tried several indexes , etc ... it will never import data.

To check if data is present I make query on localhost:9200 or I browse Kibana search bar "Index name or pattern" with the index "call_samples".

Can anyone help me on this please ? I'm stuck at this point ... Thanks

EDIT 2 : Ok I'm dumb on this one, I just wanted to redirect the logs streams to a file when I was launching logstash :

logstash.bat -f ..\conf\logstash.conf -v > logfile.txt

But it was probably breaking the input file from being imported. So I just removed the part where I redirect on file :

logstash.bat -f ..\conf\logstash.conf -v

Now, my index is correctly created, but no data is being imported ...

If you've run this more than once, logstash has remembered the earlier run and sees nothing new to process. You'll need to find your registry file (.sincedb*) and edit/remove it. — Alain Collins
@Alain Collins oh yeah I forgot to say that I regulary delete the sincedb files created on my User context directory (C:\Users\myusername), so I think it see changes :/ Or maybe there are stored in another location too ? — Alex

Alex Alex · Accepted Answer · 2016-02-11T10:51:20

It was an encoding issue, and even in verbose mode, it never told me it was failing or something ... not even a little clue.

So I tested with a new test file encoded in UTF-8 and it worked well ...

Issue importing CSV file with Logstash

1 Answers