Tensorflow crushes the JVM on amazon EMR 5.16

Question

When trying to run tensorflow inside Apache Spark on Amazon EMR, the tensorflow java library crushes the JVM on boot up with the following error:

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f7570e5f6d7, pid=24374, tid=0x00007f757e146700

JRE version: OpenJDK Runtime Environment (8.0_171-b10) (build 1.8.0_171-b10) Java VM: OpenJDK 64-Bit Server VM (25.171-b10 mixed mode linux-amd64 >compressed oops) Problematic frame: C [libtensorflow_framework.so+0x9276d7] nsync::nsync_mu_lock(nsync::nsync_mu_s_*)+0x17

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

An error report file with more information is saved as: /mnt/yarn/usercache/hadoop/appcache/application_1536299931720_0001/container_1536299931720_0001_01_000002/hs_err_pid24374.log

If you would like to submit a bug report, please visit: http://bugreport.java.com/bugreport/crash.jsp The crash happened outside the Java Virtual Machine in native code. See problematic frame for where to report the bug.

Please help if you have encountered similar issues

@Hanyun Gong Do you have an issue raised with TensorFlow, and if so, could you point me to it? — Aseem Savio

K F K F · Accepted Answer · 2020-09-23T03:19:48

I met similar problem when I run model via spark on yarn. I loaded ModelBundle in driver and broadcast it to executors to run, then I got the error. Then I loaded the model in the executors by loading model in mapPartitions function like:

val result = df.rdd
  .mapPartitions { iter =>
    val m = SavedModelBundle.load("xx/xx", "serve")
    val session = m.value.session()

    val ret = iter
      .map{ row =>
        ... ...
        val runner = session.runner()
        // run model here
        ... ...
      }.toSeq
    session.close()
    m.close()
  }

Then you can get result you want.

In my case, I have to run the model by broadcasting it to executor, but unfortunately, none of my trial succeeded. I guess some states are set in native lib, and broadcast function only send java object to executors without native states. So it ends up with native error. I'm still searching methods to broadcast and run the model.

Tensorflow crushes the JVM on amazon EMR 5.16

1 Answers