Amazon Kinesis Firehose as a cloudwatch logs consumer

3
votes

My requirement is to collect the VPC flow logs from AWS. I have created a VPC flow log and associated it to cloud watch log group "Flow log group". From CloudWatch logs subscription I thought of subscribing it through the AWS Kinesis firehose, but in the subscription list I could not find the "Kinesis Firehose", Only "Lambda" and "ElasticSearch Service". Could you please help me on this.

Cannot we subscribe the CloudWatch logs using AWS Kinesis firehose.

1
amazon-web-servicesamazon-kinesisamazon-kinesis-firehoseamazon-cloudwatchlogs
I believe you need to have the stream created before hand to associate the cloudwatch data to it – John
Hi Jonhn, Thanks for the quick reply. I have tried it by creating the kinesis steam first then I have created the VPC flow logs. After that I tried to subscribe the AWS cloudwatch log group through kinesis firehose but in AWS cloud watch console I could not have option to subscribe the logs through the kinesis firehose. – Raju
Any luck on this? I am in the same boat. – justin.m.chase

1 Answers

7
votes

A Firehose arn is a valid subscription destination for CloudWatch Logs, but it is not possible to set one with the console, only with API or CloudFormation.

Most examples I have found use the console's LogGroup option 'Stream to AWS Lambda' to feed a AWS Lambda that forwards to Amazon Kinesis Firehose, such as in How to Visualize and Refine Your Network’s Security by Adding Security Group IDs to Your VPC Flow Logs.

The Using CloudWatch Logs Subscription Filters chapter of the Amazon CloudWatch Logs User Guide has a Firehose Example (Example 3) that might be a better fit for your use case.

Essentially, the path is Amazon CloudWatch LogGroup SubscriptionFilter -> Amazon Kinesis Firehose Delivery Stream -> Amazon S3 Bucket and Prefix

Fun Fact: Amazon CloudWatch Logs data is already GZIPped when delivered to Firehose.