Query Python logs in Glue Query with CloudWatch Insights

Question

I'm very new to CloudWatch Insights, and I've been trying to understand how to get it to work with Python logging. Currently I have an AWS Glue ETL query setup in PySpark/Python. I am using the default logging package for Python in the script.

I've read the documentation, and I couldn't find any details as to how to format the logging that would make it query-able through CloudWatch Insights. Ideally, I would like to setup different fields in the log messages that I can query and get the values from through with Insights.

Here's an example of a logging message in the script:

import timeit

start = timeit.default_timer()

...run some code

stop = timeit.default_timer()

runtime = stop - start

logging.info('Runtime: {}'.format(runtime))

I would want to query the custom field like @Runtime to show all the runtimes in that column for different runs. With this, I would also like to see a simple Insight query example so I can build on that.

Anyone help would be really appreciated!

Can you put in the bits of code you have till now... The logging is the same as the python logging we usually write in python. What is the format you are looking for.. Examples would be helpful — Emerson

Emerson Emerson · Accepted Answer · 2020-04-06T00:09:04

Its the sames as setting up a simple logger

Simple sample below

MSG_FORMAT = '%(asctime)s %(levelname)s %(name)s: %(message)s'
logging.basicConfig(format=MSG_FORMAT)
logger = logging.getLogger('Something')
logger.setLevel(logging.INFO)

Then your code

start = timeit.default_timer()

...run some code

stop = timeit.default_timer()

runtime = stop - start

logger.info('Runtime: {}'.format(runtime))

Query Python logs in Glue Query with CloudWatch Insights

1 Answers