ElasticSearch FileBeat or LogStash SysLog input recommendation

1
votes

I'm planning to receive SysLog data from various network devices that I'm not able to directly install beats on and trying to figure out the best way to go about it. I know Beats is being leveraged more and see that it supports receiving SysLog data, but haven't found a diagram or explanation of which configuration would be best practice moving forward. FileBeat looks appealing due to the Cisco modules, which some of the network devices are. To break it down to the simplest questions, should the configuration be one of the below or some other model?

  • Network Device > LogStash > Elastic
  • Network Device > LogStash > FileBeat > Elastic
  • Network Device > FileBeat > Elastic
  • Network Device > FileBeat > LogStash > Elastic

We want to have the network data arrive in Elastic, of course, but there are some other external uses we're considering as well, such as possibly sending the SysLog data to a separate SIEM solution. I know we could configure LogStash to output to a SIEM but can you output from FileBeat in the same way or would this be a reason to ultimately send to LogStash at some point?

Thank you in advance!

1
elasticsearchlogstashfilebeat

1 Answers

2
votes

Our SIEM is based on elastic and we had tried serveral approaches which you are also describing. At the end we're using Beats AND Logstash in between the devices and elasticsearch. This is why: Our infrastructure is large, complex and heterogeneous. The Filebeat syslog input only supports BSD (rfc3164) event and some variant. Using the mentioned cisco parsers eliminates also a lot. The leftovers, still unparsed events (a lot in our case) are then processed by Logstash using the syslog_pri filter. And finally, forr all events which are still unparsed, we have GROKs in place.

Besides the syslog format there are other issues: the timestamp and origin of the event. Some events are missing any timezone information and will be mapped by hostname/ip to a specific timezone, fixing the timestamp offsets. Other events have very exotic date/time formats (logstash is taking take care). Other events contains the ip but not the hostname. In this cases we are using dns filter in logstash in order to improve the quality (and thaceability) of the messages.

I my opinion, you should try to preprocess/parse as much as possible in filebeat and logstash afterwards. Without logstash there are ingest pipelines in elasticsearch and processors in the beats, but both of them together are not complete and powerfull as logstash. And if you have logstash already in duty, there will be just a new syslog pipeline ;)

Edit:

Finally there is your SIEM. With Beats your output options and formats are very limited. Filebeat also limits you to a single output. In Logstash you can even split/clone events and send them to different destinations using different protocol and message format.