How can i merge or Concatenate data frame having non equal column number in spark

Question

I am doing a project using spark. in some stage i need to merge or concatenate 3 data frame in single data frame. these data frame is coming from spark sql table i have used union function which already merge column with same number from two table but i need to merge unequal column values too . I am confused now is there any way to merging or concatenating unequal column based data frame in pyspark kindly guide me

Answer 1

You could add a column with a default value before merging.

from pyspark.sql.functions import lit

updDf = df2.withColumn('zero_column', lit(0))

df1.union(updDf)