How to call a hive UDF written in Java using Pyspark from Hive Context

2
votes

I use getLastProcessedVal2 UDF in hive to get the latest partitions from table. This UDF is written in java . I would like to use the same UDF from pyspark using hive context.

dfsql_sel_nxt_batch_id_ini=sqlContext.sql(''' select l4_xxxx_seee.**getLastProcessedVal2**("/data/l4/work/hive/l4__stge/proctl_stg","APP_AMLMKTE_L1","L1_AMLMKT_MDWE","TRE_EXTION","2.1")''')

Error:

ERROR exec.FunctionRegistry: Unable to load UDF class: java.lang.ClassNotFoundException:

2
hadoopapache-sparkpysparkapache-spark-sql

2 Answers

1
votes

start your pyspark shell as:

pyspark --jars /path/to.udf.jar <all-other-param>

OR

submit your pyspark job with --jars option as:

spark-submit --jars /path/to/udf.jar <all-other-param>

-1
votes

You could register that user defined function using SQLContext method udf, there you can see that you have to pass a string as the first parameter and it will represent the name of your udf while using SQL queries.

e.g.

sqlContext.udf().register("slen",
       (String arg1) -> arg1.length(),
       DataTypes.IntegerType);

sqlContext.sql("SELECT slen(name) FROM user").show();