Pyspark Cast StructType as ArrayType<StructType>

Question

I have a dataframe df1 with a column col1 that has structure :

StructField(recipientResource,ArrayType(StructType(List(StructField(resourceId,StringType,true),StructField(type,StringType,true))),true),true)

and another dataframe df2 with col1 that has structure:

StructField(recipientResource,StructType(List(StructField(resourceId,StringType,true),StructField(type,StringType,true))),true)

Inorder to union df1.union(df2), I was trying to cast the column in df2 to convert it from StructType to ArrayType(StructType), however nothing which I tried has worked out.

Can anyone suggest how to go about the same. I'm new to pyspark, any help is appreciated.

array<struct<...>> and struct<...> are two completely different objects - you cannot cast one into another. You could add wrapping array if that's what you mean, like select(array(struct_column)). — Alper t. Turker
An minimal reproducible example with a small sample of your dataframes and the desired output would be helpful. See more on how to create good reproducible apache spark dataframe examples. — pault

Sourya_cool Sourya_cool · Accepted Answer · 2020-07-26T13:28:04

Here is a simple solution using array() function:

Input:

df1 (with ArrayType(StructType()) column):

df2 (with StructType() column):

Code:

df2=(df2
     .withColumn('recipientResource',array(col('recipientResource'))) #convert StructType() column to ArrayType(StructType()) column
    )

Output:

Modified df2:

df3 (output after union of df1 and df2):

Pyspark Cast StructType as ArrayType<StructType>

1 Answers