Filtering out data in Spark dataframe in Scala

Question

I have a dataframe df, which contains below data:

**customers**   **product**   **Val_id**
     1               A            1
     2               B            X
     3               C               
     4               D            Z

I have successfully filtered for data where column val_id is blank

df.where(col("val_id").isin(""))

But I am not able to figure out a way to filter data where column val_id is not blank, i tried something like below, but did not work for me:

df.where(col("val_id").isnotin(""))

Can anyone please help me to achieve it using Spark Scala.

Answer 1

You can use filter to get desired output:

df.filter("rule_id != ''")

Answer 2

Assuming Val_id is of String type, you can use this inequality operator !==:

df.where(col("Val_id") !== "").show

Conversely, you can also use === for matching the blank.

Answer 3

If column type is String:

df.where(trim(col("val_id")) != "")