Fastest way to check if DataFrame(Scala) is empty?

3
votes

How to check if DataFrame(Scala) is empty in fastest way?I use DF.limit(1).rdd.isEmpty, faster than DF.rdd.isEmpty,but not ideal.Is there any better way to do that?

1
scalaapache-sparkapache-spark-sqlspark-dataframe
Does this answer your question? How to check if spark dataframe is empty? – user3370741

1 Answers

3
votes

I usually wrap a call to first around a Try:

import scala.util.Try

val t = Try(df.first)

From there you can match on it if it's a Success or Failure to control logic:

import scala.util.{Success,Failure}

t match {
  case Success(df) => //do stuff with the dataframe

  case Failure(e) => 
    // dataframe is empty; do other stuff
    //e.getMessage will return the exception message
}