Spark Scala UDF : java.lang.UnsupportedOperationException: Schema for type Any is not supported

Question

I am trying to return a map from the UDF with in if else and getting the below exception , Any pointers please ?

java.lang.UnsupportedOperationException: Schema for type Any is not supported

import org.apache.spark.sql.functions.{col, udf}
import org.apache.spark.sql.functions._
val df2  = Seq(
  ("1", Map("Fld1" -> "USA","Fld2" -> "UK")),
  ("2", Map("Fld1" -> "Germany", "Fld2" -> "Portugal")),
("3", Map("Fld1" -> "France", "Fld2" -> "Paris"))
).toDF("id", "map")

val getmapUdf  = udf((map1: Map[String, String]) => { 
   
    val fl1 = map1.getOrElse("Fld1","unknown")   
    val fl2 = map1.getOrElse("Fld2","unknown")   

     if (fl1 =="Germany")
    {
            Map("key1" -> "G")
    }
    else if(fl1 =="France") 
    {
        if (fl2 =="UK")
        {
            Map("key1" ->"U")
        }
        else
        {
            Map("key1" ->"Y")
        }
    }
    else if(fl1 =="France") 
    {
           Map("key1" ->"G")
    }
    
})
var temp2 = df2.withColumn("mymap", getmapUdf($"map"))
temp2.show(false)

Answer 1

You get that error because your UDF function doesn't always return a type Map[String,String], you are using if/else statements that do not cover default values when conditions are not satisfied, so the return type is Any.

However, you can do the same thing w/o UDF actually, using when function:

var temp2 = df2.withColumn(
  "mymap",
  when($"map" ("Fld1") === "Germany", map(lit("key1"), lit("G")))
    when ($"map" ("Fld1") === "France" && $"map" ("Fld2") === "UK", map(lit("key1"),lit("G")))
    when ($"map" ("Fld1") === "France", map(lit("key1"), lit("Y")))
)

temp2.show(false)

//+---+-----------------------------------+-----------+
//|id |map                                |mymap      |
//+---+-----------------------------------+-----------+
//|1  |[Fld1 -> USA, Fld2 -> UK]          |null       |
//|2  |[Fld1 -> Germany, Fld2 -> Portugal]|[key1 -> G]|
//|3  |[Fld1 -> France, Fld2 -> Paris]    |[key1 -> Y]|
//+---+-----------------------------------+-----------+

Anyway if you want to use UDF, modify the function to return Option[Map[String,String]]. Something like this:

val getmapUdf = udf((map1: Map[String, String]) => {
  val fl1 = map1.getOrElse("Fld1", "unknown")
  val fl2 = map1.getOrElse("Fld2", "unknown")

  if (fl1 == "Germany") {
    Some(Map("key1" -> "G"))
  } else if (fl1 == "France") {
    if (fl2 == "UK") {
      Some(Map("key1" -> "U"))
    } else {
      Some(Map("key1" -> "Y"))
    }
  } else if (fl1 == "France") {
    Some(Map("key1" -> "G"))
  } else {
    None
  }
})

Answer 2

I will just try to provide an alternative answer here if you want to keep using what you are doing but @blackbishop answer cover most of the options covered to implement the same stuff.

In order for your code to work you can just make a change in your UDF to make sure you have else condition that returns a default map as shown below and then you won't get that error.

val getmapUdf  = udf((map1: Map[String, String]) => { 
   
    val fl1 = map1.getOrElse("Fld1","unknown")   
    val fl2 = map1.getOrElse("Fld2","unknown")   

     if (fl1 =="Germany")
    {
            Map("key1" -> "G")
    }
    else if(fl1 =="France") 
    {
        if (fl2 =="UK")
        {
            Map("key1" ->"U")
        }
        else
        {
            Map("key1" ->"Y")
        }
    }
    else if(fl1 =="France") 
    {
           Map("key1" ->"G")
    }
  else
  {
    Map("key1" -> "unknown")
  }
    
})