Unable to use Flink RocksDB state backend due to endianness mismatch

0
votes

My Flink Job reads from a kafka topic and store the data in a RocksDB state backend, in order to make use of the queryable state. I'm able to run the job and query the state in my local machine. But when deploying on the cluster, I'm getting the below error:

Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:900)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:843)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:843)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: java.lang.IllegalStateException: Could not initialize keyed state backend.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:286)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:199)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:666)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:654)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:257)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.Exception: Could not load the native RocksDB library
at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:483)
at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:235)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:785)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:277)
... 6 more

Caused by: java.lang.UnsatisfiedLinkError: /data/hadoop/tmp/rocksdb-lib-32bc6a5551b331596649309a808b287d/librocksdbjni-linux64.so: /data/hadoop/tmp/rocksdb-lib-32bc6a5551b331596649309a808b287d/librocksdbjni-linux64.so: ELF file data encoding not big-endian (Possible cause: endianness mismatch)
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:460)

I have tried setting the rocksDB state backend at cluster level and job level. when it is set at job level, I have provided it as a shaded dependency. I have tried compiling the code on the host cluster machine as well. I get the same error in all the cases. 

ELF file data encoding not big-endian (Possible cause: endianness mismatch)

How can I solve this error?

1
apache-flinkflink-streamingrocksdb
Are you trying to run RocksDB on a server with an ARM processor? If so, see this thread: github.com/facebook/rocksdb/issues/678 Sounds like it might be possible to make it work. – David Anderson
@alpinegizmo my server is IBM Power7 (ppc64 architecture), and I'm using Flink 1.2. Flink manages the rocksDB state backend internally, right? How can a user configure it then? I'm not familiar with it. – mmziyad
I'm no expert on rocksdb, but since rocksdb is a native (JNI) library, I think you'll have to find or build a JAR that contains rocksdb compiled for PPC, and then provide that jar file to Flink. I'm also seeing that PPC support was very recently fixed in rocksdb -- apparently version 5.3.6 fixed the rocksdb java build for PPC just 15 days ago. – David Anderson
thanks for the inputs @alpinegizmo. I could solve the issue by building frocksdb with the patch you mentioned. – mmziyad

1 Answers

1
votes

Solved the issue by:

  1. clone frocksdb.
  2. Apply the rocksdb patch from this commit.
  3. run make rocksdbjava on the target platform.
  4. Add the resulting rocksdbjni-<version-platform>.jar as a local maven dependency
  5. build flink on target platform, with frocksdbjni in flink-statebackend-rocksdb/pom.xml pointing to the local maven jar.

Note: This patch is already available in latest version of rocksdb. Once frocksdb is updated to the current version of rocksdb, this issue should not occur.