Java application with Apache Livy

Question

I decided to build a web service(app) for Apache Spark with Apache Livy.
Livy server is up and running on localhost port 8998 according to Livy configuration defaults.
My test program is a sample application in Apache Livy documentation: https://livy.incubator.apache.org/docs/latest/programmatic-api.html

While creating LivyClient by LivyClientBuilder class,

 client = new LivyClientBuilder().setURI(new 
 URI("http","user:info","localhost",8998,"","",""))
    .build();

I got "URI is not supported by any registered client factories" exception:

Exception in thread "main" java.lang.IllegalArgumentException: URI 'http://%5Bredacted%5D@localhost:8998?#' is not supported by any registered client factories.
at org.apache.livy.LivyClientBuilder.build(LivyClientBuilder.java:155)
at Client.<init>(Client.java:17)
at Client.main(Client.java:25)

I found out client instance stays null in LivyClientBuilder class.

client = factory.createClient(uri, this.config);

factory is an instance of LivyClientFactory interface.
The only class which implements the interface is RSCClientFactory.
In RSCClientFactory we have this piece of code:

if (!"rsc".equals(uri.getScheme())) {
     return null;
 }

I've tried "rsc" instead of "http", this is the error:

    2018-09-15 11:32:55 ERROR RSCClient:340 - RPC error.
  java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: Client closed before SASL negotiation finished.
  javax.security.sasl.SaslException: Client closed before SASL negotiation finished.
    at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
    at org.apache.livy.rsc.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:419)
    at org.apache.livy.rsc.JobHandleImpl.get(JobHandleImpl.java:60)
    at org.apache.livy.rsc.rpc.SaslHandler.channelInactive(SaslHandler.java:92)
    at Client.main(Client.java:39)

Apache Livy is running on http://localhost:8998 then I think we need submit our jar file to this address, but I don't understand "rsc" there.

I would appreciate if anyone guides me about these problems.

Alex Alex · Accepted Answer · 2018-10-09T17:10:03

You just need to pass your URL as a String:

LivyClient client = new LivyClientBuilder()
                .setURI(new URI("http://localhost:8998"))
                .build();

After that you can add your *.jar file with:

client.addJar("file://...yourPathToJarHere.../*.jar");

or

client.uploadJar(new File("...."));

It depends on your cluster configuration. You can find full java API description here: https://livy.incubator.apache.org/docs/latest/api/java/index.html

Java application with Apache Livy

2 Answers