Log retrieval solution using AWS CloudWatch insights logs

Question

I'm planning to implement a log retrieval solution from AWS CloudWatch logs using Insights logs query which let the users get the logs for 15 days. The queried data can range any where between KBs to GBs for that time frame.

Is there any way to retrieve that log data in paginated way using AWS services? Since the response limit is capped for API Gateway and Lambda, it is hard to retrieve the data in paginated fashion.

Are there any other AWS services that can be used to retrieve the cloudwatch logs data?

Answer 1

One option is that first you can export Cloudwatch data to S3. You may export data periodically from Cloudwatch or use Lambda subscription to export data to s3.

Once data are in s3, you can use Athena to query data using SQL like language.