Filebeat logs to Logstash connection

Question

No matter what I do, I cannot seem to make filebeat forward logs to logstash as a deb service(/etc/init.d/logstash start). When I run the logstash executable in my terminal pointing to the config file this connection seems to work. There has to be something slightly off with my configuration or understanding of how to run logstash as a detached service.

Environment

2 AWS hosted ubuntu servers, no security group and/nacls blocking the connection

Filebeats installed on my application servers

filebeat.yml passes configtest
filebeat stays running as a service

Logstash installed as deb on a master server

logstash.conf passes config test and is saved in /etc/logstash/conf.d
Elasticsearch is commented out in logstash.conf
logstash is running as a service
logstash.conf Permissions-> -rw-rw-rw- logstash logstash logconsolidated.log
No cert/ssl/tls keys/encryption are being used at this time

Filebeat config

filebeat:
 # List of prospectors to fetch data.
      prospectors:
          paths:
            - /opt/somedirectoy/test/log/mylog.log
          input_type: log
  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["10.10.10.10:5044"]

Logstash.conf

input {
  beats {
    port => 5044
  }
}

output {
  file {
    path => "/opt/logstash/output/log-consolidated.log"
    }
  }

In order to understand more about your logstash as a service, please check the file /etc/init.d/logstash. Post this file here. You can open it as a text file and see what directory has been specified as default config directory. Sometimes we need to edit this file to suit our requirements. — Mrunal Pagnis

Tw K Tw K · Accepted Answer · 2016-07-01T04:18:59

Can you check the worker count in filebeat.yml as below?

  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["10.10.10.10:5044"]
    # Number of workers per Logstash host.
    worker: 1

You should add worker count in the logstash configuration

Filebeat logs to Logstash connection

1 Answers