AWS Glue Python ETL: logger messages appear in the error cloudwatch stream

Question

I'm writing a Glue ETL, and I'm trying to log using Python default's logger.

The problem is that all the log messages I'm printing using the logger appear in the error stream of the job.

If I print directly to stdout (using print), I see the printed messages in the regular cloudwatch log stream.

I tried to redirect my logger to stdout, but I still got the same result: messages appear in error stream.

Does anyone know how I can use logger, and still see my messages in the log cloudwatch stream? (and not in the error cloudwatch stream)

This is the code sample I'm using to test:

import logging
import sys

MSG_FORMAT = '%(asctime)s %(levelname)s %(name)s: %(message)s'
DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=MSG_FORMAT, datefmt=DATETIME_FORMAT, stream=sys.stdout)
logger = logging.getLogger()

logger.setLevel(logging.INFO)

logger.info("Test log message. This appear on the job error cloudwatch stream")

print("This is a print. This appear on the job log cloudwatch stream")

jfhfhf839 jfhfhf839 · Accepted Answer · 2020-06-03T13:51:48

I ended up adding

logging.getLogger().addHandler(logging.StreamHandler(sys.stdout))

to my logger definition. That resolved the problem

AWS Glue Python ETL: logger messages appear in the error cloudwatch stream

1 Answers