How to troubleshoot logging to cloudwatch from docker in an ECS container?

Question

I'm tearing my hair out over this. Here's my setup:

ECS Container on an EC2 Instance that contains an
ECS Task Definition that runs a
Docker instance with a python script that logs to stderr

I see a Cloudwatch log group fo the ECS Task get created, but nothing I print to stderr appears.

My ECS container has the default ecsInstanceRole. The ecsInstanceRole has the AmazonEC2ContainerServiceforEC2Role policy which is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:CreateCluster",
                "ecs:DeregisterContainerInstance",
                "ecs:DiscoverPollEndpoint",
                "ecs:Poll",
                "ecs:RegisterContainerInstance",
                "ecs:StartTelemetrySession",
                "ecs:UpdateContainerInstancesState",
                "ecs:Submit*",
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
                ],
                "Resource": "*"
            }
        ]
    }

Task definition

{
    "containerDefinitions": [
        {
            "dnsSearchDomains": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "torres",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "torres"
                }
            },
            "command": [
                "sleep",
                "infinity"
            ],
            "cpu": 100,
            "mountPoints": [
                {
                    "readOnly": null,
                    "containerPath": "/mnt/efs",
                    "sourceVolume": "efs"
                }
            ],
            "memoryReservation": 512,
            "image": "X.dkr.ecr.us-west-2.amazonaws.com/torres-docker:latest",
            "interactive": true,
            "essential": true,
            "pseudoTerminal": true,
            "readonlyRootFilesystem": false,
            "privileged": false,
            "name": "torres"
        }
    ],
    "family": "torres",
    "volumes": [
        {
            "name": "efs",
            "host": {
                "sourcePath": "/mnt/efs"
            }
        }
    ]
}

Can you share your task Task Definition details as well?. Have you made sure that ECS agent has awslogs driver enabled. ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs"] . Also, please make sure that your application(python) is logging to console. — Imran
Thanks for responding Imran. I added the task definition. The app was logging to the console, the ECS agent is version 1.22 so it doesn't need that config as per here docs.aws.amazon.com/AmazonECS/latest/developerguide/… — sshevlyagin

sshevlyagin sshevlyagin · Accepted Answer · 2018-12-03T13:29:45

I figured out the issue, but I don't know why it solves it. When I changed my command from sleep infinity to just running a piece of code that logs python script.py the logging in CloudWatch started working.

Before I was launching the container with sleep infinity ssh-ing into the container and launching script.py from the shell and this was NOT logging.

How to troubleshoot logging to cloudwatch from docker in an ECS container?

1 Answers