Serialize error when storm supervisor starts executor

0
votes

I have an apache storm nimbus and an supervisor running separately. Then I submitted the topology(jars containing code of building topology, spout, bolt and the dependency classes) to storm by using the storm jar command. After that on the supervisor host, I found the exception show below:

2016-09-06 21:49:32.649 org.apache.storm.daemon.executor [INFO][main] Loading executor __acker:[8 8]
2016-09-06 21:49:32.661 org.apache.storm.daemon.executor [INFO][main] Loaded executor tasks __acker:[8 8]
2016-09-06 21:49:32.672 org.apache.storm.daemon.executor [INFO][main] Timeouts disabled for executor __acker:[8 8]
2016-09-06 21:49:32.673 org.apache.storm.daemon.executor [INFO][main] Finished loading executor __acker:[8 8]
2016-09-06 21:49:32.688 org.apache.storm.daemon.executor [INFO][main] Loading executor EventDistributionBolt:[2 2]
2016-09-06 21:49:32.711 org.apache.storm.daemon.executor [INFO][main] Loaded executor tasks EventDistributionBolt:[2 2]
2016-09-06 21:49:32.713 org.apache.storm.daemon.executor [INFO][main] Finished loading executor EventDistributionBolt:[2 2]
2016-09-06 21:49:32.728 org.apache.storm.daemon.executor [INFO][main] Loading executor SubscriptionFinderBolt:[6 6]
2016-09-06 21:49:32.735 org.apache.storm.daemon.executor [INFO][main] Loaded executor tasks SubscriptionFinderBolt:[6 6]
2016-09-06 21:49:32.742 org.apache.storm.daemon.executor [INFO][main] Finished loading executor SubscriptionFinderBolt:[6 6]
2016-09-06 21:49:32.753 org.apache.storm.daemon.executor [INFO][main] Loading executor __system:[-1 -1]
2016-09-06 21:49:32.755 org.apache.storm.daemon.executor [INFO][main] Loaded executor tasks __system:[-1 -1]
2016-09-06 21:49:32.757 org.apache.storm.daemon.executor [INFO][main] Finished loading executor __system:[-1 -1]
2016-09-06 21:49:32.768 org.apache.storm.daemon.executor [INFO][main] Loading executor __metricscom.xxx.alp.storm.metric.ServiceLogMetricsConsumer:[10 10]
2016-09-06 21:49:32.771 org.apache.storm.daemon.executor [INFO][main] Loaded executor tasks __metricscom.xxxx.alp.storm.metric.ServiceLogMetricsConsumer:[10 10]
2016-09-06 21:49:32.777 org.apache.storm.daemon.executor [INFO][main] Finished loading executor __metricscom.xxxx.alp.storm.metric.ServiceLogMetricsConsumer:[10 10]
2016-09-06 21:49:32.789 org.apache.storm.daemon.executor [INFO][main] Loading executor EventStreamSpout:[4 4]
2016-09-06 21:49:32.807 org.apache.storm.messaging.netty.Client [ERROR][client-boss-1] connection attempt 2 to Netty-Client-alp-storm-supervisor-2a-i-f478ab2f.XXXXXXXX.com/xxxx:6700 failed: java.net.ConnectException: Connection refused: alp-storm-supervisor-2a-i-f478ab2f.XXXX.com/xxxx:6700
2016-09-06 21:49:32.936 org.apache.storm.daemon.worker [ERROR][main] Error on initialization of server mk-worker
java.lang.RuntimeException: java.io.InvalidClassException: com.fasterxml.jackson.databind.deser.BasicDeserializerFactory; local class incompatible: stream classdesc serialVersionUID = -1426550576764902820, local class serialVersionUID = 2445376702910286321
        at org.apache.storm.utils.Utils.javaDeserialize(Utils.java:179) ~[storm-core-1.0.x.jar:?]
        at org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:430) ~[storm-core-1.0.x.jar:?]
        at org.apache.storm.daemon.task$get_task_object.invoke(task.clj:74) ~[storm-core-1.0.x.jar:?]
        at org.apache.storm.daemon.task$mk_task_data$fn__5625.invoke(task.clj:177) ~[storm-core-1.0.x.jar:?]
        at org.apache.storm.util$assoc_apply_self.invoke(util.clj:930) ~[storm-core-1.0.x.jar:?]
        at org.apache.storm.daemon.task$mk_task_data.invoke(task.clj:170) ~[storm-core-1.0.x.jar:?]
        at org.apache.storm.daemon.task$mk_task.invoke(task.clj:181) ~[storm-core-1.0.x.jar:?]
        at org.apache.storm.daemon.executor$mk_executor$fn__5844.invoke(executor.clj:371) ~[storm-core-1.0.x.jar:?]
        at clojure.core$map$fn__4553.invoke(core.clj:2622) ~[stormjar.jar:?]
        at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[stormjar.jar:?]
        at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[stormjar.jar:?]
        at clojure.lang.RT.seq(RT.java:507) ~[stormjar.jar:?]
        at clojure.core$seq__4128.invoke(core.clj:137) ~[stormjar.jar:?]
        at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[stormjar.jar:?]
        at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) ~[stormjar.jar:?]
        at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) ~[stormjar.jar:?]
        at clojure.core$reduce.invoke(core.clj:6519) ~[stormjar.jar:?]
        at clojure.core$into.invoke(core.clj:6600) ~[stormjar.jar:?]
        at org.apache.storm.daemon.executor$mk_executor.invoke(executor.clj:372) ~[storm-core-1.0.x.jar:?]
        at org.apache.storm.daemon.worker$fn__6474$exec_fn__2320__auto__$reify__6476$iter__6481__6485$fn__6486.invoke(worker.clj:634) ~[storm-core-1.0.x.jar:?]
        at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[stormjar.jar:?]
        at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[stormjar.jar:?]
        at clojure.lang.Cons.next(Cons.java:39) ~[stormjar.jar:?]
        at clojure.lang.RT.next(RT.java:674) ~[stormjar.jar:?]
        at clojure.core$next__4112.invoke(core.clj:64) ~[stormjar.jar:?]
        at clojure.core$dorun.invoke(core.clj:3010) ~[stormjar.jar:?]
        at clojure.core$doall.invoke(core.clj:3025) ~[stormjar.jar:?]
        at org.apache.storm.daemon.worker$fn__6474$exec_fn__2320__auto__$reify__6476.run(worker.clj:634) ~[storm-core-1.0.x.jar:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_102]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_102]
        at org.apache.storm.daemon.worker$fn__6474$exec_fn__2320__auto____6475.invoke(worker.clj:606) ~[storm-core-1.0.x.jar:?]
        at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[stormjar.jar:?]
        at clojure.lang.AFn.applyTo(AFn.java:144) ~[stormjar.jar:?]
        at clojure.core$apply.invoke(core.clj:630) ~[stormjar.jar:?]
        at org.apache.storm.daemon.worker$fn__6474$mk_worker__6569.doInvoke(worker.clj:580) [storm-core-1.0.x.jar:?]
        at clojure.lang.RestFn.invoke(RestFn.java:512) [stormjar.jar:?]
        at org.apache.storm.daemon.worker$_main.invoke(worker.clj:768) [storm-core-1.0.x.jar:?]
        at clojure.lang.AFn.applyToHelper(AFn.java:165) [stormjar.jar:?]
        at clojure.lang.AFn.applyTo(AFn.java:144) [stormjar.jar:?]
        at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.0.x.jar:?]
Caused by: java.io.InvalidClassException: com.fasterxml.jackson.databind.deser.BasicDeserializerFactory; local class incompatible: stream classdesc serialVersionUID = -1426550576764902820, local class serialVersionUID = 2445376702910286321
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) ~[?:1.8.0_102]
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) ~[?:1.8.0_102]
        at org.apache.storm.utils.Utils.javaDeserialize(Utils.java:175) ~[storm-core-1.0.x.jar:?]
        ... 39 more
2016-09-06 21:49:32.957 org.apache.storm.util [ERROR][main] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
        at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.x.jar:?]
        at clojure.lang.RestFn.invoke(RestFn.java:423) [stormjar.jar:?]
        at org.apache.storm.daemon.worker$fn__6474$mk_worker__6569.doInvoke(worker.clj:580) [storm-core-1.0.x.jar:?]
        at clojure.lang.RestFn.invoke(RestFn.java:512) [stormjar.jar:?]
        at org.apache.storm.daemon.worker$_main.invoke(worker.clj:768) [storm-core-1.0.x.jar:?]
        at clojure.lang.AFn.applyToHelper(AFn.java:165) [stormjar.jar:?]
        at clojure.lang.AFn.applyTo(AFn.java:144) [stormjar.jar:?]
        at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.0.x.jar:?]

From the exception, I figured out it was because the version of jackson-databind in the uploaded jar was not the same as the jackson-databind the supervisor uses. Then I make them the same and re-upload the jars again. The executor at the supervisor node could be started without issue this time.

Based on this issue, my questions are:

  1. Do I have to make all the shared classes between supervisor and the jar I uploaded the same version? This seems impossible, since once I uploading new jars containing new version or deploying new version of class to the supervisor, the issue above would happen.
  2. What is the serialize/deserialize machanism storm uses during starting an executor inside the worker process? What would it serialize/deserialize?
  3. If the answer to the first question is yes, how to upgrade a class shared by supervior and the uploaded jar without impacting the stream processing?
1
apache-storm

1 Answers

1
votes

From the error, it appears that the error is more specifically because of the unique code used to serialize serialVersionUID = -1426550576764902820, local class serialVersionUID = 2445376702910286321.

Typically, your jackson databind of the supervisor and the one you use locally should not have to match. I use jackson databind for Jongo in my code and I never bother that the version matches with the supervisor. As long as they are in different namespaces, so there should not be any conflict.