How Scala dataframe filtering OUT?

0
votes

I have a dataframe df, which like below data:

|  path |dst|
|1->2->3| 2 |
|1->2->3| 4 |

Now I want to filter out where dst isin path, So the result df I want as below:

|  path |dst|
|1->2->3| 4 |

I had try:

df.filter(v=>v.getAs[String]("path").split("->").contains(v.getAs[String]("dst")))

which return the df that path contain dst, and then I want to use not:

df.filter(!(v=>v.getAs[String]("path").split("->").contains(v.getAs[String]("dst")))
df.filter(~(v=>v.getAs[String]("path").split("->").contains(v.getAs[String]("dst")))

but Idea is red under my code. So what can I do?

2
scaladataframeapache-sparkapache-spark-sql

2 Answers

2
votes

Here is a solution

val df = Seq("1->2->3" -> "2", "1->2->3" -> "4").toDF("path", "dst") 
val dstNotInPath = df.filter(!array_contains(split($"path", "->"), $"dst"))

dstNotInPath.show

dstNotInPath: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [path: 
string, dst: string]
+-------+---+
|   path|dst|
+-------+---+
|1->2->3|  4|
+-------+---+

Now, let's learn how to fish:

  1. create a suitable dataframe (it wasn't mentioned in your question)
  2. define what you want to do, with typing:
  • split a string column into an array of strings
  • test if this array contains the value of another column
  • invert test (and filter)
  1. look on internet of each of these operations
  • split string into array -> I found the split method, and an example of how to use it
  • test if array contains: found array_contains and an example
  • invert test was as you mentioned

Note:

  • Here I use the "dataframe api", where I declare symbolically what operations on the dataframe columns need to be done. This is recommended for such operations (filtering on a type Dataframe = Dataset[Row] objects)
  • You use the "rdd api" where you apply a scala function on each Row type entry of the dataframe. It means that the function is serialized, send to each worker, and executed there on the java/scala Row instances.
1
votes

You can write it like this:

df2.filter(v=> !v.getAs[String]("path").split("->").contains(v.getAs[String]("dst")))

Or solution with DataFrame API:


    df.withColumn("splitted_path",functions.split($"path", "->"))
      .withColumn("filter_c",array_contains($"splitted_path", $"dst"))
      .where(!$"filter_c")
      .drop("splitted_path","filter_c")