How to see Python print statements from running Fargate ECS task?

Question

I have a Fargate ECS container that I use to run a Docker container through tasks in ECS. When the task starts, an sh script is called, runner.sh,

#!/bin/sh
echo "this line will get logged to ECS..."
python3 src/my_python_script.py # however print statements from this Python script are not logged to ECS

This in turn starts a long-running Python script, my_python_script.py. I know the Python script is running fine because it does what it needs to do, but I can't see output from the Python script.

Inside of my_python_script.py there are several print() statements. In the CloudWatch logs for my ECS Fargate task, I see output from the sh script ("this line will get logged to ECS..."), but not output from print() statements that are made within the Python script.

This is the logs configuration from inside my task definition:

{
    "ipcMode": null,
    "executionRoleArn": "myecsTaskExecutionRolearn",
    "containerDefinitions": [
        {
            "dnsSearchDomains": null,
            "environmentFiles": null,
            "logConfiguration": {
                "logDriver": "awslogs",
                "secretOptions": null,
                "options": {
                    "awslogs-group": "/ecs/mylogsgroup",
                    "awslogs-region": "eu-west-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "entryPoint": null,
            "portMappings": [],
            "command": null,
            "linuxParameters": null,
            "cpu": 0,
            "environment": [],
            "resourceRequirements": null,
            "ulimits": null,
            "dnsServers": null,
            "mountPoints": [],
            "workingDirectory": null,
            "secrets": null,
            "dockerSecurityOptions": null,
            "memory": null,
            "memoryReservation": null,
            "volumesFrom": [],
            "stopTimeout": null,
            "image": "1234567.dck.aws.com/mydockerimage",
            "startTimeout": null,
            "firelensConfiguration": null,
            "dependsOn": null,
            "disableNetworking": null,
            "interactive": null,
            "healthCheck": null,
            "essential": true,
            "links": null,
            "hostname": null,
            "extraHosts": null,
            "pseudoTerminal": null,
            "user": null,
            "readonlyRootFilesystem": null,
            "dockerLabels": null,
            "systemControls": null,
            "privileged": null,
            "name": "my-task-definition-name"
        }
    ],
    "memory": "4096",
    "taskRoleArn": "myecsTaskRolearn",
    "family": "my-task-definition-name",
    "pidMode": null,
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "networkMode": "awsvpc",
    "cpu": "2048",
    "inferenceAccelerators": [],
    "proxyConfiguration": null,
    "volumes": [],
    "tags": []
}

Dockerfile:


FROM rocker/verse:3.6.0
ENV DEBIAN_FRONTEND noninteractive

RUN install2.r --error \
    jsonlite

RUN echo "deb http://ftp.de.debian.org/debian testing main" >> /etc/apt/sources.list
RUN echo 'APT::Default-Release "stable";' | tee -a /etc/apt/apt.conf.d/00local
RUN apt-get update && apt-get -t testing install -y --force-yes python3.6
RUN apt-get update && apt-get -t testing install -y libmagick++-dev python3-pip python-setuptools 

RUN mkdir /app
WORKDIR /app
COPY ./src /app/src

RUN pip3 install --trusted-host pypi.python.org -r /app/requirements.txt

CMD /app/runner.sh

I think I am following the awslogs instructions from https://docs.aws.amazon.com/AmazonECS/latest/userguide/using_awslogs.html but maybe not? Is there something obvious I need to do to make sure that print() statements from within a Python script are captured in my ECS task's CloudWatch logs?

Can you post the rest of the task definition and the Dockerfile that built the image? — Alexandre Juma
@alexandre-juma I have edited my question to add my task definition and Dockerfile. Thanks — Benjamin
Can you try changing your Dockerfile CMD statement to exec format CMD ["/app/runner.sh"] and add change your helper script like exec python3 src/my_python_script.py ? After this, please log into the container interactively and check if PID 1 is held by your python script just to make sure. — Alexandre Juma
This is a long shot but maybe the logging driver is freaking out because your PID 1 is a wrapper shell (you're not running CMD in exec mode), then calling another shell script /app/runner.sh and then calling the python interpreter and script. — Alexandre Juma
The accepted answer worked but thank you for your help @AlexandreJuma — Benjamin

evangineer evangineer · Accepted Answer · 2020-10-13T19:09:09

Seems to me that there are a couple of things you could be dealing with here.

The first is the default buffering behaviour of Python, which could stop the output from showing up. You will need to stop this.

You can set the PYTHONUNBUFFERED env var correctly by inserting the following before CMD:

ENV PYTHONUNBUFFERED=1

Secondly, quoting from the Using the awslogs driver doc that you linked:

The type of information that is logged by the containers in your task depends mostly on their ENTRYPOINT command. By default, the logs that are captured show the command output that you would normally see in an interactive terminal if you ran the container locally, which are the STDOUT and STDERR I/O streams. The awslogs log driver simply passes these logs from Docker to CloudWatch Logs. For more information on how Docker logs are processed, including alternative ways to capture different file data or streams, see View logs for a container or service in the Docker documentation.

So going by that, I would replace the CMD line with the following as per the Exec form of ENTRYPOINT:

ENTRYPOINT ["/app/runner.sh"]

This should serve to hook up the STDOUT and STDERR I/O streams for your shell script and hopefully your Python script to the container logging.

How to see Python print statements from running Fargate ECS task?

1 Answers