Logstash csv importing

Question

I'm using Ubuntu 14.04 LTS, Kibana, Logstash and Elasticsearch. I tried the following code to import my csv file to LogStash but it doesnt detect.

input 
{
    file 
    {
        path => "/home/kibana/Downloads/FL_insurance_sample.csv"
        type => "FL_insurance_sample.csv"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

filter 
{
    csv 
    {
    columns => ["policyID","statecode","country","eq_site_limit","hu_site_limit",
        "fl_sitelimit","fr_site_limit","tiv_2011","tiv_2012","eq_site_deductible",
        "hu_site_deductible","fl_site_deductible","fr_site_deductible","point_latitude",
        "point_longtitude","line","construction","point_granularity"]
        separator => ","
    }
}

output 
{
    elasticsearch {
        action => "index"
        host => "localhost"
        index => "promosms-%{+dd.MM.YYYY}"
        workers => 1
    }
    stdout
    {
        codec => rubydebug
    }

}

I even did

sudo service logstash restart

When I went into index mapping in Kibana GUI interface, i chose Logstash-* and couldn't find the data that I wanted. P.S. my config file is stored in /etc/logstash/conf.d/simple.conf

Can you start your logstash on the command line and increase the debug level with bin/logstash --debug -f your_config.conf and edit your question with the output you get? — Val
how do I start my logstash? I only did sudo service logstash restart because I can't seem to run -f command. Can I know the exact command for running it? I tried cd /etc/logstash/ and ran /etc/logstash -f logstash.conf — imbadatcoding
You can also put your logstash config file in /etc/logstash/conf.d/logstash.conf when running as a service and it will be picked up. — Val
Do you mean restarting the service? I can't seem to use the -f command line to run the config file though :/ — imbadatcoding
When running logstash as a service, the /etc/logstash/conf.d/logstash.conf config file is automatically picked up. — Val

Alcanzar Alcanzar · Accepted Answer · 2015-08-13T14:48:55

In your question, you state that you went to Logstash-* in Kibana, but your configuration file says that you are putting data into promosms-%{+dd.MM.YYYY}.

You need to go into kibana4's setup section and put [promosms-]DD.MM.YYYY into the Index name or pattern box and check both the "index contains time-based events" and "Use event times to create index names".

Then you might also want to set that as your default index.

Logstash csv importing

1 Answers