Forward onpremise logs to Kinesis Firehose

Question

For my application logging I am planning to forward the log from my on-premise servers to AWS Kinesis firehose. I am trying to use Logstash and the log-stash-output-plugin https://github.com/samcday/logstash-output-kinesis

But I think this plugin requires to forward the log data to Kinesis Data Stream and then we can create a pipe to Kinesis Firehose. If we point to the Kinesis Firehose Delivery stream then we are getting the error - ResourceNotFoundException Message: Stream test under account XXXX not found

But if we point the log-stash to Kinesis data stream and then pipe it to Kinesis Firehose then it is working fine.

Have anyone used this log-stashoutput plugin to forward the log data directly to Kinesis Firehose.

Answer 1

logstash-output-kinesis uses Kinesis Producer library(KPL) to write data to Kinesis streams. As of now KPL does not support writing to Firehose directly. - https://github.com/awslabs/amazon-kinesis-producer/issues/29

Also Firehose and Kinesis have different service endpoints and in logstash-output-kinesis project it seems there is only the option to set the kinesis endpoint. So even if you specify a valid existing Firehose name in the config, since the endpoint set is that of AWS kinesis, you should expect to see "ResourceNotFoundException". - https://github.com/samcday/logstash-output-kinesis/blob/acffae5beddcf00aa599c1fa75428f57bb7e7e30/lib/logstash/outputs/kinesis.rb#L55