Type mismatch error on checking presence of element in a Set in Spark Scala API

Question

I have defined following function written in Scala, for finding out whether particular type of nodes(Project,Filter) is contained in a given set or not:

private val operators = Set(Project.getClass,Filter.getClass)    
def containsNode(plan: Seq[LogicalPlan]):Boolean=
    {
     for(p<- plan)
       {
         if(operators.contains(p.getClass))
           true
       }
  false
}

When running the code I am getting the following error for the above function:

Error:(182, 36) type mismatch; found : Class[T(in value $anonfun)] where type T(in value $anonfun) <: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan

required: Class[_ >: T(in value operators) with T <: Serializable]
if(operators .contains(p.getClass))

I thought p.getClass would give same type of value that set operators expect. But it doesn't seem like it. I am new to Scala and Spark, so I am not sure what exactly is going on here. How can I fix this error?

Alexey Romanov Alexey Romanov · Accepted Answer · 2019-12-08T14:06:01

In this case the problem is too precise type inference, you can fix it by explicitly saying operators is a set of any classes:

private val operators = Set[Class[_]](Project.getClass,Filter.getClass)

Though a second problem is that just using true like that won't return it as you probably intend. You can explicitly write return true but better to use exists which already contains the desired logic:

def containsNode(plan: Seq[LogicalPlan]): Boolean =
    plan.exists(p => operators.contains(p.getClass))

EDIT: a third possible problem is that Project in Project.getClass is not the class, but its companion object, you probably want classOf[Project] instead.

Type mismatch error on checking presence of element in a Set in Spark Scala API

1 Answers