I have been using Cloudwatch Logstash plugin to stream Lambda application logs from Cloudwatch to Logstash. But lately it is taking more thank 6 hours for some of the logs to be pulled into logstash. I have checked logstash logs and there are no errors, but I can see .sincedb file is lagging behind. Elasticsearch support suggested that since that isn't an official plugin, so I should go with AWS module for Filebeat which is a module for aws logs. It uses filebeat s3 input to get log files from AWS S3 buckets with SQS notification. The documentation is available at https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-aws.html.
But according to AWS documentation about exporting application logs from Cloudwatch into S3
Log data can take up to 12 hours to become available for export
But I want to export data in real time. I have raised this in AWS support forums, but I am not getting any reply. Can someone please suggest a way to stream application logs from Cloudwatch into Logstash in real time
