log to specific logstream in cloudwatch from lambda

19
votes

I'd like to log to a specific CloudWatch log stream from lambda.

Currently, each lambda that fires off logs to a uniquely-named log stream, and often creates a new log stream as needed. This is problematic if I fire off 10 lambdas simultaneously with different events to process and want to look at the particular log stream for a given event type (that I could hypothetically define beforehand).

I'd like to be able to create a log stream in the console and specify to log to that particular log stream from within my lambda (python code), ideally without resorting to the boto3 module (although that's fine if it's the only way, of course).

enter image description here

1
amazon-web-servicesaws-lambda
You're not going to have much luck trying to do this because even if you manually call AWS.CloudWatchLogs.pushLogEvents() to send logs to a specific log stream from your Lambda function, you're limited to 5 requests/second/log-stream. Which is exactly why Lambda automatically creates multiple log streams for you instead of writing to a single log stream per Lambda function. – idbehold
If you'd like to have some search capabilities on the logs, pipe it to elastic search cluster. That works great. – johni
@johni, any reason to not just log directly to elastic search from my lambda instead of going through cloudwatch? – Alex
Yes. A lambda function should do the minimum necessary IO operations and do just what it's meant for. So for logging, you just console.log inside your lambda because AWS took care of the rest. – johni
Hmm - why wouldn't use the "Search Log Group" option that appears above all the streams for a given log group? This will show the aggregate activity across the streams (and you let you search.) – scolestock

1 Answers

9
votes

This is a comment from @scolestock... It doesn't answer the question exactly (which I was hoping I'd find) but is a great alternative that helped me.

Use the "Search Log Group" option that appears above all the streams for a given log group? This will show the aggregate activity across the streams (and you let you search.)

That's basically what I need... viewing all logs across all streams.