AWS Cloudwatch logs not working as expected

Question

I am trying to use AWS CloudWatch to maintain the application logs in a Ubuntu EC2 instance. I have installed the awslogs agent using the following command as suggested in their documentation to monitor the file application.log and push any new entries in the file to CloudWatch.

Setup command - sudo python3 ./awslogs-agent-setup.py --region ap-south-1

It was working fine for a day when I tested it out after setting it up, then it stopped working from the next day. I can see that the changes in the log files are being detected by the AWS Agent, as there is an entry in the awslogs.log file as soon as there is a new entry in the application.log file. However, the same updates are not being pushed/reflected in the CloudWatch console.

What might have gone wrong here?

Entry in /var/log/awslogs.log

2020-02-27 12:19:03,376 - cwlogs.push.reader - WARNING - 1388 - Thread-4 - Fall back to previous event time: {'end_position': 10483213, 'timestamp': 1582261391000, 'start_position': 10483151}, previousEventTime: 1582261391000, reason: timestamp could not be parsed from message.
2020-02-27 12:19:07,437 - cwlogs.push.publisher - INFO - 1388 - Thread-3 - Log group: branchpayout-python-pilot, log stream: ip-172-27-99-136_application.log, queue size: 0, Publish batch: {'fallback_events_count': 2, 'source_id': 'c0bd7124acf1c35ede963da6b8ec9882', 'num_of_events': 2, 'first_event': {'end_position': 10483151, 'timestamp': 1582261391000, 'start_position': 10482278}, 'skipped_events_count': 0, 'batch_size_in_bytes': 985, 'last_event': {'end_position': 10483213, 'timestamp': 1582261391000, 'start_position': 10483151}}

Configuration in /var/awslogs/etc/awslogs.conf

[/home/ubuntu/application-name/application.log]
file = /home/ubuntu/application-name/application.log
datetime_format = %Y-%m-%d %H:%M:%S,%f
log_stream_name = {hostname}_application.log
buffer_duration = 5000
log_group_name = branchpayout-python-pilot
initial_position = end_of_file
multi_line_start_pattern = {datetime_format}

Have you followed Step 1: Configure Your IAM Role or User for CloudWatch Logs in the doc you link? From the first look it seems like instance doesn't have proper permissions. Don't forget to attach role to the instance. — Oleksii Donoha

mahendra rathod mahendra rathod · Accepted Answer · 2020-10-23T12:11:42

Check you log format and accordingly update your awslogs.conf. for me nginx access log format in access.log was "%d/%b/%Y:%H:%M:%S %z" hence my config file contains :

datetime_format = %d/%b/%Y:%H:%M:%S %z

Below are the example .

Nginx error.log 2017/08/12 05:04:00 %Y/%m/%d %H:%M:%S
Nginx access.log    12/Aug/2017:06:19:17 +0900  %d/%b/%Y:%H:%M:%S %z
php-fpm error.log   12-Aug-2017 05:24:38    %d-%b-%Y %H:%M:%S
php-fpm www-error.log   10-Aug-2017 23:40:46 UTC    %d-%b-%Y %H:%M:%S
messages    Aug 12 06:13:36 %b %d %H:%M:%S
secure  Aug 11 04:03:33 %b %d %H:%M:%S

AWS Cloudwatch logs not working as expected

1 Answers