Stream AWS CloudWatch Log Group to Multiple AWS Elasticsearch Services

Question

Is there a way to stream an AWS Log Group to multiple Elasticsearch Services or Lambda functions?

AWS only seems to allow one ES or Lambda, and I've tried everything at this point. I've even removed the ES subscription service for the Log Group, created individual Lambda functions, created the CloudWatch Log Trigger, and I can only apply the same CloudWatch Log trigger on one Lambda function.

Here is what I'm trying to accomplish:

CloudWatch Log Group ABC -> No Filter -> Elasticsearch Service #1

CloudWatch Log Group ABC -> Filter: "XYZ" -> Elasticsearch Service #2

Basically, I need one ES cluster to store all logs, and another to only have a subset of filtered logs.

Is this possible?

Answer 1

I've ran into this limitation as well. I have two Lambda's (doing different things) that need to subscribe to the same CloudWatch Log Group.

What I ended up using is to create one Lambda that subscribes to the Log Group and then proxy the events into an SNS topic.

Those two Lambdas are now subscribed to the SNS topic instead of the Log Group.

For filtering events, you could implement them inside the Lambda.

It's not a perfect solution but it's a functioning workaround until AWS allows multiple Lambdas to subscribe to the same CloudWatch Log Group.

Answer 2

I was able to resolve the issue using a bit of a workaround through the Lambda function and also using the response provided by Kannaiyan.

I created the subscription to ES via the console, and then unsubscribed, and modified the Lambda function default code.

I declared two Elasticsearch endpoints:

var endpoint1 = '<ELASTICSEARCH ENDPOINT 1>';
var endpoint2 = '<ELASTICSEARCH ENDPOINT 2>';

Then, declared an array named "endpoint" with the contents of endpoint1 and endpoint2:

var endpoint = [endpoint1, endpoint2];

I modified the "post" function which calls the "buildRequest" function that then references "endpoint"...

function post(body, callback) {
  for (index = 0; index < endpoint.length; ++index) {
    var requestParams = buildRequest(endpoint[index], body);
...

So every time the "post" function is called it cycles through the array of endpoints.

Then, I modified the buildRequest function that is in charge of building the request. This function by default calls the endpoint variable, but since the "post" function cycles through the array, I renamed "endpoint" to "endpoint_xy" to make sure its not calling the global variable and instead takes the variable being inputted into the function:

function buildRequest(endpoint_xy, body) {
  var endpointParts = endpoint_xy.match(/^([^\.]+)\.?([^\.]*)\.?([^\.]*)\.amazonaws\.com$/);
...

Finally, I used the response provided by Kannaiyan on using the AWS CLI to implement the subscription to the logs, but corrected a few variables:

aws logs put-subscription-filter \
--log-group-name <LOG GROUP NAME> \
--filter-name <FILTER NAME> 
--filter-pattern <FILTER PATTERN> 
--destination-arn <LAMBDA FUNCTION ARN>

I kept the filters completely open for now, but will now code the filter directly into the Lambda function like dashmug suggested. At least I can split one log to two ES clusters.

Thank you everyone!

Answer 3

Seems like AWS console limitation,

You can do it via command line,

aws logs put-subscription-filter \
    --log-group-name /aws/lambda/testfunc \
    --filter-name filter1 \
    --filter-pattern "Error" \
    --destination-arn arn:aws:lambda:us-east-1:<ACCOUNT_NUMBER>:function:SendToKinesis

You also need to add permissions as well.

Full detailed instructions,

http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html

Hope it helps.

Answer 4

As of September 2020 CloudWatch now allows two subscriptions to a single CloudWatch Log Group, as well as multiple Metric filters for a single Log Group.

Update: AWS posted October 2, 2020, on their "What's New" blog that "Amazon CloudWatch Logs now supports two subscription filters per log group".