Terraform: cloudwatch logs to elasticsearch

0
votes

I am trying to push the cloudwatch logs to elastic search either using a Lambda function or Amazon Kinesis. I have the log groups setup and the elastic search domain running using terraform. Please suggest on how can I push the logs from the log group to elastic search. Please share if you have the terraform codes for the same.

1
elasticsearchlambdaterraform
Please specify what have you tried already, what's working, what's not, etc. to expect an answer. - Manish Kumar Sharma
Thanks for responding. I am looking for guidance on how/what is best. I was able to do manually "stream to amazon elastic search" from the console and in the interim setting up the IAM role (default) and also Amazon provides the default Lambda function as well. With that I am pushing the logs to elastic search. Now, I am not sure how/which terraform modules should be used to set it up. - Bond
I have a module written to handle this; however it's closed-source. I'm looking to get this approved to open-source it soon and will follow-up with a link when I do. - TJ Biddle
Thanks @TJBiddle - Bond

1 Answers

1
votes

This answer documents some example Terraform code for creating a lambda and Cloudwatch subscription that ships logs from a Cloudwatch log group to a Sumologic HTTP collector (just a basic HTTP POST endpoint). The Cloudwatch subscription invokes the Lambda every time a new batch of log entries is posted to the log group.

The cloudwatch-sumologic-lambda referred to in that Terraform code was patterned off of the Sumologic Lambda example.

I'd imagine you would to do something similar, but re-writing the Lambda to format the HTTP however ElasticSearch requires. I'd bet some quick googling on your part will turn up plenty of examples.

Alternatively to all this Terraform config though, you can just go to your Cloudwatch console, select the log group you're interested in and select "Stream to Amazon ElasticSearch".

enter image description here

Though I think that will only work if you're using the AWS "ElasticSearch service" offering - meaning if you installed/configured ElasticSearch on some EC2 instances yourself it probably won't work.