I have AWS GLUE Python application by default logs are available in Cloudwatch Log Group /aws-glue/jobs/output and /aws-glue/jobs/error for stdout and stderr respectively. I have explored AWS documentations and several websites for the process to redirect logs to a custom Cloudwatch Log Group like /dev/<app_name>/, but no luck. If any one has idea over this, please share the process.
2
votes
Do you use a Python script or so, to perform the job? And also please tell more about what you've tried so that people would not come up with what you've already tried.
– vahdet
Yes using a Python script. Unfortunately until now could not find any option to achieve this. I'm worrying even if this is possible.
– Tarun Mootala
Couldn't find any direct option. Workaround used cloudwatch event (glue job state change) with Lambda target. Segregated the logs based on event and job name, saved to s3.
– Tarun Mootala
2 Answers
0
votes
You can achieve this by passing a parameter to the Glue job:
'--continuous-log-logGroup': 'custom_log_group_name'
More details on the docs.
0
votes
for spark type glue jobs you can enable continuous logging but for a python shell type glue job continuous logging is not available , therefore you will not be able to create a custom log group for that, but there is a workaround for this. refer below for more info
How to use a CloudWatch custom log group with Python Shell Glue job?
