How to insert AWS Cloudwatch logs into database?

Question

I enabled logs for API Gateway in Cloudwatch. I can see the logs in the Cloudwatch console.

Now I need to maintain a database that can store the information which contains the request method, request id, response body, and other necessary information when the API gateway method is being invoked.

So that I can query to that database to see what methods are accessed in API gateway and when it is accessed and need to display it in an angular application.

I researched the same and found nothing.

What I think: We need to use some AWS native service that scheduled for every 2 or 5 mins to read the logs of the API gateway and insert into DynamoDb or any other Database. But don't know how to use.

Please advise how to insert API Gateway logs into the database. Thanks in advance

rather then sending logs to database you send database to s3 and it will be cost efficient way also and low memory. you can check this out for more details docs.aws.amazon.com/AmazonCloudWatch/latest/logs/… ......aws.amazon.com/blogs/aws/… — Harsh Manvar

pradeep pradeep · Accepted Answer · 2019-01-03T09:12:45

Storing AWS Cloudwatch logs in a Database (like MySQL/Aurora) can be expensive. The cheapest way is to store logs in AWS S3 and then maybe run Spark Jobs(AWS EMR) to analyze your data.

If you are looking for AWS native service, then you can look for AWS ElasticSearch service and here is good documentation from AWS on how to stream Cloudwatch Logs to AWS ElasticSearch -https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html

Once the data is in Elasticsearch you can query and build nice visualization in Kibana dashboard.

Hope this help.

How to insert AWS Cloudwatch logs into database?

1 Answers