Spark DataFrameWriter ignoreNullFields not working

Question

I have a DataFrame containing multiple nulls with different schema

df.show(false)
+----+----+----+----+
|col1|col2|col3|col4|
+----+----+----+----+
|null|null|1   |a   |
+----+----+----+----+

I am trying to write this dataframe to HDFS as a JSON file but Spark omits the fields that are null while writing the JSON. This is understandable as ignoreNullFields is set to true by default

But even when I use

spark.write.option("ignoreNullFields", "false").json(...)

or

spark.write.option("ignoreNullFields", false).json(...)

Columns containing null values get omitted.

Is there something that I'm missing while using the ignoreNullFields option?

Answer 1

You're not using the ignoreNullFields option correctly

the correct way is :

df.coalesce(1).write.mode('overwrite').json(ignoreNullFields=False,path="a")

to keep also the columns that contain only null values

Answer 2

works when you put it as a configuration setting in the SparkSession

spark.conf.set("spark.sql.jsonGenerator.ignoreNullFields", false)