How to configure logstash and filebeat SSL communication

Question

The question:

Can someone help me figure out why I can't get filebeats to talk to logstash over TLS/SSL?

The Error:

I can get the filebeat and logstash to talk to eachover with TLS/SSL disabled, but when i enable it and use the settings/config below, I get the following error (observed in logstash.log):

{:timestamp=>"2016-10-28T17:21:44.445000+0100", :message=>"Pipeline aborted due to error",
 :exception=>java.lang.NullPointerException, :backtrace=>["org.logstash.netty.PrivateKeyCo
nverter.generatePkcs8(org/logstash/netty/PrivateKeyConverter.java:43)", "org.logstash.nett
y.PrivateKeyConverter.convert(org/logstash/netty/PrivateKeyConverter.java:39)", "java.lang
.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "RUBY.create_server(/usr/share
/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.0.beta4-java/lib/logstash/
inputs/beats.rb:139)", "RUBY.register(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/log
stash-input-beats-3.1.0.beta4-java/lib/logstash/inputs/beats.rb:132)", "RUBY.start_inputs(
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:311)", "org.jruby.RubyArray.eac
h(org/jruby/RubyArray.java:1613)", "RUBY.start_inputs(/usr/share/logstash/logstash-core/li
b/logstash/pipeline.rb:310)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/lo
gstash/pipeline.rb:187)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/pipelin
e.rb:145)", "RUBY.start_pipeline(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:2
40)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:error}
{:timestamp=>"2016-10-28T17:21:47.452000+0100", :message=>"stopping pipeline", :id=>"main"
, :level=>:warn}
{:timestamp=>"2016-10-28T17:21:47.456000+0100", :message=>"An unexpected error occurred!",
:error=>#<NoMethodError: undefined method `stop' for nil:NilClass>, :backtrace=>["/us
r/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.0.beta4-java/lib/lo
gstash/inputs/beats.rb:173:in `stop'", "/usr/share/logstash/logstash-core/lib/logstash/inp
uts/base.rb:88:in `do_stop'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logst
ash/logstash-core/lib/logstash/pipeline.rb:366:in `shutdown'", "/usr/share/logstash/logsta
sh-core/lib/logstash/agent.rb:252:in `stop_pipeline'", "/usr/share/logstash/logstash-core/
lib/logstash/agent.rb:261:in `shutdown_pipelines'", "org/jruby/RubyHash.java:1342:in `each
'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:261:in `shutdown_pipelines'",
 "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:123:in `shutdown'", "/usr/share/
logstash/logstash-core/lib/logstash/runner.rb:237:in `execute'", "/usr/share/logstash/vend
or/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logsta
sh/logstash-core/lib/logstash/runner.rb:157:in `run'", "/usr/share/logstash/vendor/bundle/
jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bo
otstrap/environment.rb:66:in `(root)'"], :level=>:fatal}

The Setup:

Servers

• 2 servers.

  $> uname -a Linux elkserver 3.10.0-327.36.2.el7.x86_64 #1 SMP Mon Oct 10 23:08:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux $> cat /etc/*-release CentOS Linux release 7.2.1511 (Core)

• SELinux is Permissive (soz).

• Firewalls are of. (mazza soz).
• One server runs elasticsearch and logstash; one runs filebeat.

Elasticsearch

$> /usr/share/elasticsearch/bin/elasticsearch -version Version: 2.4.1, Build: c67dc32/2016-09-27T18:57:55Z, JVM: 1.8.0_111

Logstash

$> /usr/share/logstash/bin/logstash -V logstash 5.0.0-alpha5

Filbeat

$> /usr/share/filebeat/bin/filebeat -version filebeat version 5.0.0 (amd64), libbeat 5.0.0

Config:

• Logstash

input {
  beats {
    port => 5044
  ssl => true
  ssl_certificate => "/etc/pki/tls/certs/filebeat-forwarder.crt"
  ssl_key => "/etc/pki/tls/private/filebeat-forwarder.key"
  }
}
output {
  elasticsearch {
  hosts => "localhost:9200"
  manage_template => false
  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  document_type => "%{[@metadata][type]}"
  }
}

• Filebeat.yml

output:
 logstash:
   enabled: true
   hosts:
     - "<my ip address>:5044"
   timeout: 15
   tls:
     certificate_authorities:
     - /etc/pki/tls/certs/filebeat-forwarder.crt
filebeat:
 prospectors:
   -
     paths:
       - /var/log/syslog
       - /var/log/auth.log
     document_type: syslog
   -
     paths:
       - /var/log/nginx/access.log
     document_type: nginx-access

• File: openssl_extras.cnf:

  [req]    
  distinguished_name = req_distinguished_name    
  x509_extensions = v3_req    
  prompt = no    
  [req_distinguished_name]    
  C = TG    
  ST = Togo    
  L =  Lome    
  O = Private company    
  CN = *    
  [v3_req]    
  subjectKeyIdentifier = hash    
  authorityKeyIdentifier = keyid,issuer    
  basicConstraints = CA:TRUE    
  subjectAltName = @alt_names        
  [alt_names]    
  DNS.1 = *    
  DNS.2 = *.*    
  DNS.3 = *.*.*    
  DNS.4 = *.*.*.*    
  DNS.5 = *.*.*.*.*    
  DNS.6 = *.*.*.*.*.*    
  DNS.7 = *.*.*.*.*.*.*    
  IP.1 = <my ip address>
  

The command used to create the cert:

$> openssl req -subj '/CN=elkserver.system.local/' -config /etc/pki/tls/openssl_extras.cnf \ -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout /etc/pki/tls/private/filebeat-forwarder.key \ -out /etc/pki/tls/certs/filebeat-forwarder.crt

Answer 1

In Filebeat 5.0 the tls configuration setting was changed to ssl to be consistent with the configuration setting used in Logstash and Elasticsearch. Try updating your Filebeat configuration.

References:

• Securing Communication With Logstash by Using SSL
• Breaking Changes in 5.0