How to add jars on hive shell in a java application

Question

i am trying to add jar on the hive shell. I am aware of the global option on the server but my requirement is to add them per session on the hive shell. I have used this class for the hdfs dfs commands to add the jars to the hdfs file system

This is what i have tried:

Created a folder on the hdfs /tmp Add the file to hdfs filesystem using FileSystem.copyFromLocalFile method (equivalent to the hdfs dfs -put myjar.jar /tmp

Set permissions on the file on fhe fs file system

Check that the jar was loaded to hdfs using the getFileSystem method

List files on the fs FileSystem using listFiles to confirm the jars are there.

This works and I have the jars loaded to hdfs but i cannot add jars to the hive session

When i am trying to add it in the hive shell, i am doing the following:

statement = setStmt(createStatement(getConnection()));
query = "add jar " + path;
statement.execute(query);

I am getting this error [For example path of /tmp/myjar.jar]:

Error while processing statement: /tmp/myjar.jar does not exist

Other permutations on the path such as

query = "add jar hdfs://<host>:<port>" + path;
query = "add jar <host>:<port>" + path;

results with an error.

command to list jars works (with no results)

query = "list jars";
ResultSet rs = statement.executeQuery(query);

hi?? have you solve this problem? are you trying to register a custom udf?? — Kenry Sanchez
there is a way to register custom functions in hive.. I'd like to tell you? — Kenry Sanchez
I was. thinking, you can use the add jar statement to add a jar function in hive due to I guess you were trying to define a new hive udf function. — Kenry Sanchez

Aviza Aviza · Accepted Answer · 2020-05-17T09:16:57

I managed to solve this issue

The process failed because of the configuration of the FileSystem. This object is where we upload the jars to, before adding them on the session.

This is how you init the FileSystem FileSystem fs = FileSystem.newInstance(conf);

The object conf should have the properties of the hive server.

In order for the process to work, I needed to set the following parameter on the Configuration property conf.set("fs.defaultFS", hdfsDstStr);

How to add jars on hive shell in a java application

1 Answers