AWS Glue Boolean Transformation

Question

I am using AWS Glue and need to transform Boolean (True and False), columns within a Redshift datawarehouse schema to a "Yes"/"No" in another Redshift schema. At present, there does not appear to be a simple way to do so in the AWS Glue GUI.

I have been following the guide here as: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-transforms-map.html

and created the function:

def ConvertBoolean(dataFrame,ColumnName):

  dataFrame["booleanTransform"] = {}  

  if dataFrame[ColumnName] == True:
    dataFrame["booleanTransform"] = "Yes"
  else:
    dataFrame["booleanTransform"] = "No"

  del dataFrame[ColumnName]

  dataFrame[ColumnName] = {}

  dataFrame[ColumnName] = dataFrame["booleanTransform"]  

  del dataFrame["booleanTransform"] 

  return dataFrame

But do not know where the function should be stored or how to pass the dynamicframe as that is not noted in the documentation example provided.

How would this be best accomplished in the pyspark code of AWS Glue?

Answer 1

Do you really have to use Glue for that? It sounds as if a simple CTAS would be more time and money efficient:

CREATE TABLE newtable 
-- you may also want to set DIST and SORTKEYs for the newtable here
AS
SELECT
  CASE my_bool_column
    WHEN TRUE THEN 'Yes'
    ELSE 'No'
  END::VARCHAR(3) as my_bool_column,
  all_other_columns
FROM oldtable;

Answer 2

If you are using redshift why don't you write a sql script that does that for you. I don't think you need to do anything with glue.

Anyway if you still need to do it using glue just use the Apache Spark DataFrame:

df.withColumn("columnName", when(df.columnName, lit('Yes').otherwise(lit('No'))

Transforming back to a DynamicDataframe can be done using fromDF() function.