CloudWatch Log group missing although CloudWatch agent is working

Question

I can't see the Log group defined by Cloud Watch agent on my EC2 instance

Also, the default log group /var/log/messages is not visible.

I can't see these logs also on root account. I have other log groups configured and visible.

I have the following setup:

Amazon Linux
AMI managed role attached to instance: CloudWatchAgentServerPolicy
Agent installed via awslogs - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
Agent started successfully No errors in /var/log/awslogs.log. Looks like working normally. Log below.
Configuration done via /etc/awslogs/config/FlaskAppAccessLogs.conf
Instance has outbound access to internet
Instance security groups allows all outbound traffic

Any ideas what to check or what can be missing?

/etc/awslogs/config/FlaskAppAccessLogs.conf:

cat /etc/awslogs/config/FlaskAppAccessLogs.conf
[/var/log/nginx/access.log]
initial_position = start_of_file
file = /var/log/nginx/access.log
datetime_format = %d/%b/%Y:%H:%M:%S %z
buffer_duration = 5000
log_group_name = FlaskApp-Frontends-access-log
log_stream_name = {instance_id}

/var/log/awslogs.log

2019-01-05 17:50:21,520 - cwlogs.push - INFO - 24838 - MainThread - Loading additional configs from /etc/awslogs/config/FlaskAppAccessLogs.conf
2019-01-05 17:50:21,520 - cwlogs.push - INFO - 24838 - MainThread - Missing or invalid value for use_gzip_http_content_encoding config. Defaulting to use gzip encoding.
2019-01-05 17:50:21,520 - cwlogs.push - INFO - 24838 - MainThread - Missing or invalid value for queue_size config. Defaulting to use 10
2019-01-05 17:50:21,520 - cwlogs.push - INFO - 24838 - MainThread - Using default logging configuration.
2019-01-05 17:50:21,544 - cwlogs.push.stream - INFO - 24838 - Thread-1 - Starting publisher for [c17fae93047ac481a4c95b578dd52f94, /var/log/messages]
2019-01-05 17:50:21,550 - cwlogs.push.stream - INFO - 24838 - Thread-1 - Starting reader for [c17fae93047ac481a4c95b578dd52f94, /var/log/messages]
2019-01-05 17:50:21,551 - cwlogs.push.reader - INFO - 24838 - Thread-4 - Start reading file from 0.
2019-01-05 17:50:21,563 - cwlogs.push.stream - INFO - 24838 - Thread-1 - Starting publisher for [8ff79b6440ef7223cc4a59f18e5f3aef, /var/log/nginx/access.log]
2019-01-05 17:50:21,587 - cwlogs.push.stream - INFO - 24838 - Thread-1 - Starting reader for [8ff79b6440ef7223cc4a59f18e5f3aef, /var/log/nginx/access.log]
2019-01-05 17:50:21,588 - cwlogs.push.reader - INFO - 24838 - Thread-6 - Start reading file from 0.
2019-01-05 17:50:27,838 - cwlogs.push.publisher - WARNING - 24838 - Thread-5 - Caught exception: An error occurred (ResourceNotFoundException) when calling the PutLogEvents operation: The specified log group does not exist.
2019-01-05 17:50:27,839 - cwlogs.push.batch - INFO - 24838 - Thread-5 - Creating log group FlaskApp-Frontends-access-log.
2019-01-05 17:50:27,851 - cwlogs.push.publisher - WARNING - 24838 - Thread-3 - Caught exception: An error occurred (ResourceNotFoundException) when calling the PutLogEvents operation: The specified log group does not exist.
2019-01-05 17:50:27,851 - cwlogs.push.batch - INFO - 24838 - Thread-3 - Creating log group /var/log/messages.
2019-01-05 17:50:27,966 - cwlogs.push.batch - INFO - 24838 - Thread-5 - Creating log stream i-0d7e533f67870ff8d.
2019-01-05 17:50:27,980 - cwlogs.push.batch - INFO - 24838 - Thread-3 - Creating log stream i-0d7e533f67870ff8d.
2019-01-05 17:50:28,077 - cwlogs.push.publisher - INFO - 24838 - Thread-5 - Log group: FlaskApp-Frontends-access-log, log stream: i-0d7e533f67870ff8d, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1546688052000, 'start_position': 0L, 'end_position': 161L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1546708885000, 'start_position': 4276L, 'end_position': 4468L}, 'source_id': '8ff79b6440ef7223cc4a59f18e5f3aef', 'num_of_events': 24, 'batch_size_in_bytes': 5068}

Status of awslogs

sudo service awslogs status
awslogs (pid  25229) is running...

IAM role policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "ec2:DescribeTags",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams",
                "logs:DescribeLogGroups",
                "logs:CreateLogStream",
                "logs:CreateLogGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameter"
            ],
            "Resource": "arn:aws:ssm:*:*:parameter/AmazonCloudWatch-*"
        }
    ]
}

This question is off-topic for stackoverflow. This site is for questions about programming. You should delete this question and post it to superuser.com or serverfault.com, where you will reach your intended audience and are much more likely to get an answer. — anothermh

Michel Michel · Accepted Answer · 2019-01-05T19:23:05

It's seems that posting a question may quickly help to find an answer.

There is additional configuration in which i have made typo:

sudo cat /etc/awslogs/awscli.conf
[plugins]
cwlogs = cwlogs
[default]
region = us-west-1

As described above the logs are delivered to us-west-1 region. I was checking us-west-2 :)

CloudWatch Log group missing although CloudWatch agent is working

1 Answers