Why are there no logstash indexes in kibana

Question

I set up ELK stack and filebeat with my ELK node as a RedHat server following the digitalocean tutorial. Kibana is up and running, but I dont see any logstash indexes when I go to configure an index pattern as logstash-*:

Unable to fetch mapping. Do you have any indices matching the pattern?

When I do a curl to see the indexes I have, they are only filebeat indexes. Filebeat should be pushing data to logstash which is listening on 5044

$curl 'localhost:9200/_cat/indices?v'
health status index               pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   filebeat-2017.01.10   5   1       3864            0      1.7mb          1.7mb 
yellow open   filebeat-2017.06.17   5   1       1848            0    740.1kb        740.1kb 
yellow open   filebeat-2017.01.18   5   1      77062            0       33mb           33mb 
yellow open   filebeat-2017.09.14   5   1       1932            0      1.1mb          1.1mb 
yellow open   filebeat-2017.01.11   5   1      19094            0      3.6mb          3.6mb 
yellow open   .kibana

You can see I only have filebeat indexes. I checked my ports are open, and My config files are correct according to the tutorial. What could be wrong? Filebeat should be sending logs from /var/log/*.log to logstash, to elasticsearch.

When I

tail /var/log/logstash/logstash.log

there is nothing in my logstash log. I've checked and logstash, filebeat, kibana, and elasticsearch are all running. Ive also done the config file test and it said it was OK:

$sudo service logstash status
logstash is running

On my ELK node, I can clearly see the port 5044 is listening:

$ netstat -tulpn | grep -i listen | grep -v tcp6
 tcp        0      0 :::5044                     :::*                        LISTEN      -

A J A J · Accepted Answer · 2017-01-18T19:22:18

Filebeat creates daily indices using a pattern of filebeat-YYYY.MM.dd so you should not expect to see logstash indices in Elasticsearch.

The Logstash configuration recommended in the Filebeat documentation writes the data to an index based on "%{[@metadata][beat]}-%{+YYYY.MM.dd}" where [@metadata][beat] defaults to the name of the beat (filebeat) unless output.logstash.index is configured in the Filebeat config. Here's the base configuration for Logstash.

input {
  beats {
    port => 5044
  }   
}   

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }   
}

You can inspect the data in those indices to see if it's what you are expected to get from filebeat with a command like:

curl http://localhost:9200/filebeat-*/_search?pretty&size=100

Why are there no logstash indexes in kibana

2 Answers