How can I debug logstash even when configuration is OK, on Windows?

Question

I have the following configuration for my logstash importing a few CSV files:

input {
  file {
    path => [
        "C:\Data\Archive_ATS_L1\2016-10-08-00-00_to_2016-10-09-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-09-00-00_to_2016-10-10-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-10-00-00_to_2016-10-11-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-11-00-00_to_2016-10-12-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-12-00-00_to_2016-10-13-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-13-00-00_to_2016-10-14-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-14-00-00_to_2016-10-15-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv"
    ]
    start_position => "beginning"    
  }
}
filter {
  csv {
      separator => ","
      columns => ["MessageCode","SourceGuid","DateTimeGenerated","Code1","Code2","Code3","Code4","LanguageCode", "AlarmSeverity", "Message", "Guid1", "Guid2", "Guid3", "Guid4", "MessageOrigin", "RequestId", "Bool1", "Bool2", "Bool3", "Bool4", "Bool5", "Bool6", "Bool7", "Bool8", "Code5", "Code6", "Bool9", "Bool10", "Bool11", "Code7"]
  }
}
output {  
    elasticsearch {
        action => "index"
        hosts => "localhost"
        index => "S2K"
        workers => 1
    }
    stdout {}
}

I launch logstash with this command line:

logstash.bat –f ..\conf\logstash.conf --verbose

Usually I see the data that's being imported into Elasticsearch in the console. But all I get this time is one line that says "Pipeline main started" and it stays like that.

How can I check from logstash if data was imported? I tried using Elasticsearch by running: curl http://localhost:9200/_aliases

This usually gives the list of indices. But the index I have in this config (called S2K) does not get listed.

I'm new to ELK so how can I check if logstash is doing it's job? Please note that I'm using Windows 7.

you can take a look to this discuss.elastic.co/t/logstash-not-reading-file-in-windows/41723 — baudsp
I think the problem might be that you have already read the files with logstash, then you'll have to change the path of the sincedb file, which saves where logstash has read files cf elastic.co/guide/en/logstash/current/… — baudsp

Anton Anton · Accepted Answer · 2021-05-03T15:54:53

To debug logstash you need to do two things: add stdout in config, and run logstash in a proper way.

1 step: Add this config in your logstash conf file (ex.: /etc/logstash/conf.d/config.conf)

output {
  stdout {
    codec => rubydebug {
      metadata => true # Here, we will print metadata in console
    }
  }
}

2 step: Run logstash to see output with command

sudo /usr/share/logstash/bin/logstash  -f /etc/logstash/conf.d/config.conf

And you will get something like this:

{
            "log" => {
        "file" => {
            "path" => "***\\spring.log"
        }
    },
        "appName" => "my-service",
      "@metadata" => {
        "ip_address" => "***",
              "type" => "_doc",
              "beat" => "filebeat",
           "version" => "7.12.0"
    },
      "log_level" => "INFO",
     "serverName" => "***",
            "pid" => "6236",
         "thread" => "main",
        "message" => "***",
    "serviceName" => "***",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "input" => {
        "type" => "log"
    },
     "@timestamp" => 2021-01-03T10:22:07.644Z,
       "@version" => "1",
          "class" => "***"
}

Finally, after debug you can run it like sudo systemctl start logstash

Hope, it would help you, this approach helped me to save my time

How can I debug logstash even when configuration is OK, on Windows?

3 Answers