AWS Glue job fails to write to Redshift

Question

I have an AWS Glue job that should write the results from a dynamic frame to a Redshift database. For that I have set up a Glue Connection to Redshift and tested it, and it works fine. However, when the job tries to write to Redshift it timeouts with

AnalysisException: 'java.lang.RuntimeException: com.amazonaws.SdkClientException: 
Unable to execute HTTP request: 
Connect to glue.eu-central-1.amazonaws.com:443 [glue.eu-central-1.amazonaws.com/52.57.157.164, 
glue.eu-central-1.amazonaws.com/35.156.122.139, 
glue.eu-central-1.amazonaws.com/3.122.136.8] failed: connect timed out;'

The code that is used for writing looks like this:

    my_conn_options = {
    "dbtable": "tbl1",
    "database": "db1",
    "aws_iam_role": "arn:aws:iam::111111111111:role/glue-role"
}

glueContext.write_dynamic_frame.from_jdbc_conf(
    frame = dynamic_frame, 
    catalog_connection = "glue-to-redshift-connection", 
    connection_options = my_conn_options, 
    redshift_tmp_dir = args["TempDir"])

Any Idea what could be causing that?

Fritz Duchardt Fritz Duchardt · Accepted Answer · 2021-01-28T12:48:13

Once you use a Glue Connection within the job, the job is executed within your VPC and external connections no longer work, e.g to glue.eu-central-1.amazonaws.com.

In order to make internal traffic possible, you need to create a VPC endpoint for Glue.

AWS Glue job fails to write to Redshift

1 Answers