Enable cloudwatch logs for kinesis firehose cloudformation

Question

I am trying to catch Cloudwatch logs for my firehose to find any errors when sending data to S3 destination. I created a cloudformation template with logging details

"CloudWatchLoggingOptions" : {
    "Enabled" : "true",
    "LogGroupName": "/aws/firehose/firehose-dev", -->firehose-dev is my firehosedeliverystream name 
    "LogStreamName" : "s3logs"
},

I have given necesary IAM permission to firehose for creating loggroupname and streamname.

{
    "Sid": "",
    "Effect": "Allow",
    "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
    ],
    "Resource": [
        "arn:aws:logs:*:*:*"
    ]
}

When i triggered the template i didnt find any of the loggroupname and streamname is created in cloudwatch logs.

But when we give same IAM permissions to AWS::Lambda resource it will automatically create a loggroupname(i.e./aws/lambda/mylambdaname) and send the logs to the that group. But why this scenario is not working for firehose ?

As a Workaround

I am manually creating AWS::Logs::LogGroup resource with name as /aws/firehose/firehose-dev and AWS::Logs::LogStream resource with name as s3logs.

And also firehose will create a loggroup name and logstream name automatically, if we configure the firehose deliverystream using console.

Can't firehose create loggroup name and logstream name automatically like aws lambda do when configured through cloudformation?

Thanks Any help is appreciated

Marcin Marcin · Accepted Answer · 2020-05-23T09:42:49

Its resource dependent. Some resources will create the log group for you, some not. Sometimes console does create them in the background. When you use CloudFormation, usually you have to do everything yourself.

In case of Firehose you can create the AWS::Logs::LogGroup and AWS::Logs::LogStream resources in CloudFormation. For example (yaml):

MyFirehoseLogGroup:
  Type: AWS::Logs::LogGroup
  Properties: 
    RetentionInDays: 1
        
MyFirehoseLogStream:      
  Type: AWS::Logs::LogStream
  Properties: 
    LogGroupName: !Ref MyFirehoseLogGroup

Then when you define your AWS::KinesisFirehose::DeliveryStream, you could reference them:

CloudWatchLoggingOptions: 
  Enabled: true
  LogGroupName: !Ref MyFirehoseLogGroup
  LogStreamName: !Ref MyFirehoseLogStream

Enable cloudwatch logs for kinesis firehose cloudformation

1 Answers