Spark SQL and using existing hive udfs

Question

I have to read existing hive udf in spark, so in spark.sql I am creating temporary function and using it, It is working fine in spark shell but in spark-submit it's failing with the error:

org.apache.hadoop.hive.metadata.HiveException : unable to execute method public static boolean com.xxx.x.x.udfs.isb_udf.evaluate(Java.lang.string) on object xxxx.udf of class xxxx with args {Java.lang.string} of size 1 ...

I also see... Caused by Java.lang.reflect.InvocationTargetException

Code sample:

spark.sql("CREATE TEMPORARY FUNCTION currency AS 'com.spark.udf.FormatCurrency'"); 
val x = spark.sql("select currency(col1) from hive_table") ;

x.show()

Above command works in spark shell but it is not working in spark submit.

Info: If I read hive table without udf it's working in spark submit. The problem probably occurs when using hive udf.

Though it is reasonably clear what you want to do, it is easier for others to help you if you include a minimal reproducible example (command that works, command that fails, exact error message that fails). Ideally also enough context for what needs to be done to run the command. — Dennis Jaheruddin
spark.sql("CREATE TEMPORARY FUNCTION currency AS 'com.spark.udf.FormatCurrency'"); val x = spark.sql("select currency(col1) from hive_table") ;x.show. Above command works in spark shell but it is not working in spark submit — Avinash

Iam Rey Iam Rey · Accepted Answer · 2019-07-15T08:31:22

One of the reason would be, if you’re using multiple jars passing to spark submit and it contains the same class that your UDF is using then the issue might occur. I have a similar case before and I removed the jars and added a single jar in —jars along with spark submit. It works for me. I don’t know about your case. Try removing multiple jars and for sure it will work.

Spark SQL and using existing hive udfs

1 Answers