Mock an ELK Beat output to Logstash with Postman

Question

I have Elastic Beats (like metricbeat and auditbeat) that send data to logstash. For testing purposes, I'd like to mock some data from the beats to the logstash input.

I'm using the logstash output plugin in the beats (yaml config file):

output.logstash:
  enabled: true
  hosts: ["127.0.0.1:5044"]

and the beats input plugin in logstash (.conf pipeline config file):

input {
    beats {
        port => "5044"
    }
}

I naïvely assumed the beats were communicating with logstash via HTTP requests so I used Postman to make a POST request to logstash, but I got an InvalidFrameProtocolException: Invalid version of beats protocol error in logstash.

I tried to inspect the packets with wireshark to see how the original packets are formatted but all I could see was TCP packets and no HTTP protocol. I guess then that the plugins don't communicate via HTTP requests.

Can someone with more expertise comment on this ? Is there a way to mock beats output data to logstash via postman ?

I don't think it is possible, the beats input does not uses HTTP, it uses the lumberjack protocol from elastic over TCP. What do you need to test? Can you give more information? — leandrojmp
@leandrojmp thanks for the information. I need to mock beats events in logstash so that I can test my logstash pipeline and the other services that are connected to logstash. I ended up using the http logstash plugin, as per my following answer. — Mathieu Rollet

Mathieu Rollet Mathieu Rollet · Accepted Answer · 2020-09-11T17:48:10

Thanks to @leandrojmp I know the beats and logstash communicate via the lumberjack protocol and not HTTP.

I ended up using the http logstash input plugin so that I can mock my beats events via postman. I didn't modify my original pipeline configuration file though, but I used the ability of logstash to load several pipeline configuration files. So I created another logstash pipeline configuration file with my http plugin and some filter to remove the http specific data like the "header" field:

input {
    stdin { }
    http {
        port => "8081"
    }
}

filter {
    mutate {
        remove_field => ["headers"]
    }
}

I can then load the two files when I want to launch logstash for testing, by placing them in the same directory:

bin/logstash -f logstash.d/*.conf

Or just the original config file when I'm not testing:

bin/logstash -f logstash.d/pipeline.conf

Mock an ELK Beat output to Logstash with Postman

1 Answers