There is a group of AWS CloudWatch Logs, inside which there are several threads. As far as I understand it, each thread is a log coming from a separate server or container. CloudWatch Log streams
I put the whole group of logs in Kinesis Firehose to deliver them to S3 Bucket. But inside Kinesis Firehose, all the logs are merged into one. How can I get these logs to the S3 storage so that each thread has its own directory?
I found a solution: 1) I modified every log in Kinesis Firehose using the Lambda function. I added an identifier to the end of the log line. And then it looks like this: Modified logs
2) I created a Lambda function with a trigger that works every time logs are written to s3 bucket. And in this function, I distribute logs to the folders I need based on the information I added to the logs earlier. I will not give the code of this lambda function. I've described the general approach and I think those who need it can figure it out.
