Unable to add remote source with Vora 1.4 and Spark controller 2.0

0
votes

I have been try to connect Vora 1.4 to HANA 1.0 SPS12 through Spark controller 2.0. Remote source can be added to HANA studio but Vora tables are invisible. When trying to refresh remote source, a class not found error appears in Spark controller error log. Following are the error messages:

*

17/05/23 10:11:46 ERROR HanaSQLContext: Failed to set up Exended Store java.lang.ClassNotFoundException: at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.spark.sql.hana.hdfs.store.HDFSStore$$anonfun$apply$2.apply(HDFSSt>ore.scala:212) at org.apache.spark.sql.hana.hdfs.store.HDFSStore$$anonfun$apply$2.apply(HDFSSt>ore.scala:187) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.sql.hana.hdfs.store.HDFSStore$.apply(HDFSStore.scala:186) at org.apache.spark.sql.hana.HanaSQLContext.getExtendedStore(HanaSQLContext.sca>la:104) at com.sap.hana.spark.core.session.Session.(SessionManager.scala:191) at com.sap.hana.spark.core.session.Session.(SessionManager.scala:166) at com.sap.hana.spark.core.session.Session$.apply(SessionManager.scala:136) at com.sap.hana.spark.core.session.SessionManager$.startNewSession(SessionManag>er.scala:72) at com.sap.hana.spark.network.CommandHandler$$anonfun$receive$3.applyOrElse(Com>mandRouter.scala:549) at akka.actor.Actor$class.aroundReceive(Actor.scala:467) at com.sap.hana.spark.network.CommandHandler.aroundReceive(CommandRouter.scala:>432) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at akka.actor.ActorCell.invoke(ActorCell.scala:487) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) at akka.dispatch.Mailbox.run(Mailbox.scala:220) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDis>patcher.scala:397) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1>339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java>:107)

*

Any one has any idea?

Thanks!

2
vora
Frank: Thanks for the suggestion. Voraodbc certainly is a way for Hana/Hadoop connection. The issue is that voraodbc only available on HANA SPS12 revsion 205 and higher. I did checked all adapters, voraodbc is not there as my version is revision 121. I also like to setup DLM and thought Spark controller is better for that purpose. Allen - Allen Cao
Could you please give some more details? Exact versions for Vora, Spark Controller, Spark, Hadoop distribution. Also, the full Spark Controller log file (from last startup to failure) and the Spark Controller config files (hana_hadoop-env.sh, hanaes-site.xml) would be helpful. If you have access to SAP ticket system, you could also open a ticket in component HAN-VO. - Frank Legler
@FrankLegler: Vora version is 1.4 patch 0, Spark Controller 2.0SP00P1, Spark 1.6.2 from Hortonworks 2.5.3 release. Spark Controller is manual installed. Also tried the Ambari deployment for Spark Controller. With Ambari installation, the controller log would not have the "Class not found error" but the result was the same - Vora tables were invisible. Here is log file from manual installation. The error would appear when remote source in Hana Studio is refreshed. - Allen Cao
@FrankLegler this is hana_hadoop-env.sh - Allen Cao

2 Answers

0
votes

I would recommend using the new VoraODBC connectivity (a.k.a. SAP HANA Wire connectivity) to connect HANA to Vora 1.4. This gives easy access to tables in the new relational engine and disk engine without installing anything in addition.

See Vora 1.4 Developer Guide chapter "7.9 Accessing SAP Vora from SAP HANA" (see https://help.sap.com/viewer/p/SAP_VORA).

0
votes

To troubleshooting the issue, I uninstalled Spark controller 2.0 and installed Spark controller 1.6. There are no more error message in Spark controller error log. But the issue remains, the Vora tables are invisible from remote source in HANA studio. There is no error in index server log on HANA side either. It seems that Vora 1.4 has serious problem with Spark controller as 1.4 is not on the any version of Spark controller compatibility matrix. I am hoping that someone already found a fix for this.

Spark controller 1.6 log:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
 INFO [main] (HanaESConfig.scala:67) - Loaded HANA Extended Store Configuration
Found Spark Libraries. Proceeding with Current Class Path
17/06/07 13:20:48 INFO Server: Starting Spark Controller
17/06/07 13:21:01 INFO CommandRouterDefault: Connecting to Vora Engine
17/06/07 13:21:01 INFO CommandRouterDefault: Initialized Router!
17/06/07 13:21:01 INFO CommandRouterDefault: Server started
17/06/07 13:21:28 INFO VoraClientFactory: returning a Vora catalog client of this Vora catalog server: bdesxi14.localdomain:20298
17/06/07 13:21:28 INFO CatalogFactory: returning a Vora catalog client of this Vora catalog server: bdesxi14.localdomain:20298
17/06/07 13:21:28 INFO CBinder: searching for compat-sap-c++.so at /opt/rh/SAP/lib64/compat-sap-c++.so
17/06/07 13:21:28 INFO CBinder: loading compat package from /opt/rh/SAP/lib64/compat-sap-c++.so
17/06/07 13:21:28 INFO CBinder: searching for libpam.so.0 at /lib64/libpam.so.0
17/06/07 13:21:28 INFO CBinder: loading libpam.so.0 from /lib64/libpam.so.0
17/06/07 13:21:28 INFO CBinder: loading library libprotobuf.so
17/06/07 13:21:28 INFO CBinder: loading library libprotoc.so
17/06/07 13:21:28 INFO CBinder: loading library libtbbmalloc.so
17/06/07 13:21:28 INFO CBinder: loading library libtbb.so
17/06/07 13:21:28 INFO CBinder: loading library libkrb5support.so
17/06/07 13:21:28 INFO CBinder: loading library libk5crypto.so
17/06/07 13:21:28 INFO CBinder: loading library libcom_err.so
17/06/07 13:21:28 INFO CBinder: loading library libkrb5.so
17/06/07 13:21:28 INFO CBinder: loading library libgsasl.so
17/06/07 13:21:28 INFO CBinder: loading library libgssapi_krb5.so
17/06/07 13:21:28 INFO CBinder: loading library libgssrpc.so
17/06/07 13:21:28 INFO CBinder: loading library libv2runtime.so
17/06/07 13:21:28 INFO CBinder: loading library libv2auth.so
17/06/07 13:21:28 INFO CBinder: loading library libv2net.so
17/06/07 13:21:28 INFO CBinder: loading library libv2catalog_connector.so