Stream log data from CloudWatch

Question

I can presumably get all log events from a CloudWatch logStream using:

    const cwl = new AWS.CloudWatchLogs();

    cwl.getLogEvents({logGroupName, logStreamName}, (err, results) => {

      for (let r of results.events) {
        console.log(r.message);
      }

    });

however, my question is - is there a way to stream the logs instead of reading all of them at once? Seems like for some log streams there could be a million events/records so might be too much data to read all at once?

I looked at the declaration file for cloudwatchlogs.d.ts: https://github.com/aws/aws-sdk-js/blob/master/clients/cloudwatchlogs.d.ts

don't see anything that can stream the logs to a client.

Have you checked this example using AWS Lambda: docs.aws.amazon.com/AmazonCloudWatch/latest/logs/… ? You can create a custom lambda function that is invoked by the log streams and where log processing / filtering can be applied. — M. F.

Dejan Peretin Dejan Peretin · Accepted Answer · 2019-07-18T05:11:57

You can't stream it directly.

Responses from getLogEvents API contain nextForwardToken and nextBackwardToken. You can use those in subsequent requests to get the next batch of events if all of them could not have been returned in a single response. See here: https://docs.aws.amazon.com/cli/latest/reference/logs/get-log-events.html

Alternatively, you can use the createExportTask API to get the data into S3 and stream it from there. Not sure if the exported data will be in a streamable format. https://docs.aws.amazon.com/cli/latest/reference/logs/create-export-task.html

Stream log data from CloudWatch

1 Answers