Replace empty string with Null Values

Question

I have a dataframe with column TotalCharges which is string type, it has some some empty values, I want null to be printed instead of those empty spaces.

Column right now

**************
|1671.6                           |
|8003.8                           |
|680.05                           |
|6130.85                          |
|1415                             |
|6201.95                          |
|                                 |
|74.35                            |
|6597.25                          |

Expected Output

|1671.6                           |
|8003.8                           |
|680.05                           |
|6130.85                          |
|1415                             |
|6201.95                          |
|Null                             |
|74.35                            |
|6597.25                          |

Answer 1

Below way will give you null for the column when String is ""

df.withColumn("TotalCharges",when($"TotalCharges"!=="",$"TotalCharges"))

And this will give you "Null" string in place:

df.withColumn("TotalCharges",when($"TotalCharges"==="","Null").otherwise($"TotalCharges"))

Answer 2

You can try something like that:

import org.apache.spark.sql.functions.{when,lit, _}
df.withColumn("TotalCharges", when(col("name") === lit(""), null).otherwi
se(col("TotalCharges")))