How to rotate logs in AWS CloudWatch?

Question

I am using the Amazon CloudWatch Log Agent (AWS documentation) to upload logs from my EC2 instances to the CloudWatch console.

For my CloudWatch Log Agent configuration on EC2, I have the following:

[general]
state_file = /var/awslogs/state/agent-state  

[applog]
file = /var/www/html/logs/applog.log
log_group_name = MyApp
log_stream_name = applog.log
datetime_format = %Y-%m-%d %H:%M:%S

My question is, how do I get the applog.log to rotate on a daily basis? In the AWS Documentation (link above), it mentions being able to configure log rotation policies, but I can't find any mention/example of how to actually do this. I've tried specifying the log_stream_name as applog_%Y-%m-%d.log, but it interprets this literally.

Any ideas or pointers in the right direction would be very welcome - thanks!

Answer 1

I don't think there is a way to rotate the log_stream_name using Amazon's CloudWatch Log Agent. The log rotation described in the documentation is related to ingesting log files that get rotated by your system, the CloudWatch Log Agent does not perform any log rotation itself.

According to the documentation the only variables allowed in the log_stream_name property are {instance_id}, {hostname} and {ip_address}

Answer 2

I think what you are asking for is log expiry. At least that is what I was looking for. And here is how you can expire logs after a certain amount of time:

1. Through sam template
2. Through console:
   1. On the AWS console dashboard, navigate to Cloudwatch>>Logs.
   2. In the table which lists all the logs, for your particular log in question:, click on "Never expire", in the "Expire Events After" column. The popup will allow you to choose the retention period

Answer 3

According to the Agent documentation, the log filename can have wildcards.

Cloudwatch Agent Docs

File can point to a specific file or multiple files (using wildcards such as /var/log/system.log*). Only the latest file is pushed to CloudWatch Logs based on file modification time.

So, you can just start writing to a new file that matches your pattern and everything should be fine.