AWS Glue Job fails with connection timeout error

0
votes

I am new to AWS Glue. I have created a job that uses two Data Catalog tables and runs simple SparkSQL query on top of them. The job fails on the Transform step with Exception

pyspark.sql.utils.AnalysisException: 'java.lang.RuntimeException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to glue.us-east-1.amazonaws.com:443 [blah] failed: connect timed out;'

JDBC source (Redshift) VPC security group has both inbound and outbound rules configured.

I have seen another post on SO about configuring VPC endpoint for Glue itself, but I don't quite understand what it should look like? Should it be and interface to glue.us-east-1.amazonaws.com:443 or something else? I am confused.

UPD: Autogenerated pyspark script

## @params: [TempDir, JOB_NAME]
args = getResolvedOptions(sys.argv, ['TempDir','JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
## @type: DataSource
## @args: [database = "redshift_catalog", redshift_tmp_dir = TempDir, table_name = "analytics_mongo_raw_conversations", transformation_ctx = "DataSource0"]
## @return: DataSource0
## @inputs: []
DataSource0 = glueContext.create_dynamic_frame.from_catalog(database = "redshift_catalog", redshift_tmp_dir = args["TempDir"], table_name = "analytics_mongo_raw_conversations", transformation_ctx = "DataSource0")
## @type: DataSource
## @args: [database = "redshift_catalog", redshift_tmp_dir = TempDir, table_name = "analytics_mongo_raw_messages", transformation_ctx = "DataSource1"]
## @return: DataSource1
## @inputs: []
DataSource1 = glueContext.create_dynamic_frame.from_catalog(database = "redshift_catalog", redshift_tmp_dir = args["TempDir"], table_name = "analytics_mongo_raw_messages", transformation_ctx = "DataSource1")
## @type: SqlCode
## @args: [sqlAliases = {"messages": DataSource1, "conversations": DataSource0}, sqlName = SqlQuery0, transformation_ctx = "Transform0"]
## @return: Transform0
## @inputs: [dfc = DataSource1,DataSource0]
Transform0 = sparkSqlQuery(glueContext, query = SqlQuery0, mapping = {"messages": DataSource1, "conversations": DataSource0}, transformation_ctx = "Transform0")
job.commit()
2
amazon-web-servicesaws-glue
I want to run complex SQL transformations, my impression was that Glue facilitates this by managing EMR. – Semant1ka

2 Answers

0
votes

You need to add a Glue Connection so that you can connect to your Redshift Cluster. You have to make sure that this Glue Connection is placed in a private subnet.

0
votes

I was able to resolve this issue, indeed there has to be a VPC endpoint. In addition to that connection should use a private subnet with NAT gateway. My initial subnet didn't have NAT.

Example of the VPC endpoint configuration in Terraform:

resource "aws_vpc_endpoint" "glue" {
  vpc_id            = var.vpc_id
  service_name      = var.glue_vpc_service_name
  vpc_endpoint_type = "Interface"

  security_group_ids = var.security_group_ids 
  subnet_ids = var.subnet_ids

  tags = { mytag = "mytag"}
}