AWS Glue - Python Shell Jobs Secret Manager Connectivity Issues

0
votes

I am using Python Shell Jobs under AWS Glue which has boto3 and a few other libraries built-in . I am facing issues trying to access the secrets manager to get credentials to my RDS instance running Mysql , the job keeps running forever without any (error/success) message nor does it time out .

Below is the simple code that runs even from my local or a lambda for Python3.7 but not in Python Shell GLUE , 

import boto3
import base64
from botocore.exceptions import ClientError

secret_name = "secret_name"
region_name = "eu-west-1"

session = boto3.session.Session()

client = session.client(
    service_name='secretsmanager',
    region_name=region_name
)

get_secret_value_response = client.get_secret_value(SecretId=secret_name)
print(get_secret_value_response)

Would be very helpful if someone could point out if anything needs to be done additionally in Python Shell jobs under AWS Glue in order to access the secret manager credentials .

2
pythonaws-glue
did you check the cloud watch logs? what is the exception/error that you are getting, can you please provide more info. Maybe the python code is missing base64 library? We are using secret manager in AWS Glue Pyspark environment, and its working perfectly. – Yuva

2 Answers

0
votes

Make sure the IAM role used by the Glue Job has the policy SecretsManagerReadWrite

Also AWSGlueServiceRole and AmazonS3FullAccess According to the documentation

0
votes

When you create a job without any VPC configuration , then glue tries to reach the secret manager through internet , if the policies allows to have internet route then we can connect to secret manager

But when a glue job is created with VPC configuration/connection then all the request are made from your VPC/subnet where the connection points to , if this is the case, make sure you have secret manager endpoint present in your route table of the subnet where glue launches the resources.

https://docs.aws.amazon.com/glue/latest/dg/setup-vpc-for-glue-access.html

https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html