I know a similar stack has been posted by others. My question is "it looks like Ignite is trying to deserialize a GridClosureProcessor (or a closure from it?) . If so, why is it doing that? I am trying to root cause this issue but none of my code is in the stack except MyCallable mentioned at the top (not actually in the stack).
No cache puts are happening in this code path unless they are internal. I mention this because of a comment on another post that "Unknown pair" can be caused by a cache put of the wrong type.
I am focusing on
Failed to deserialize object [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
Here's the rest.
[2019-04-08 22:20:23,724][ERROR][pub-#63][GridJobWorker] Failed to initialize job [jobId=800890ff961-7ff6a786-9d4d-43d8-91a0-70225c5e3a4a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.obfucorp.aa.project.core.jobs.MyCallable, dep=GridDeployment [ts=1554761996013, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, clsLdrId=730290ff961-8d93b961-09f2-48c3-bd2f-49db31aae61e, userVer=0, loc=true, sampleClsName=o.a.i.i.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1, pendingUndeploy=false, undeployed=false, usage=1], taskClsName=com.obfucorp.aa.project.core.jobs.MyCallable, sesId=700890ff961-7ff6a786-9d4d-43d8-91a0-70225c5e3a4a, startTime=1554762023663, endTime=9223372036854775807, taskNodeId=7ff6a786-9d4d-43d8-91a0-70225c5e3a4a, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=7ff6a786-9d4d-43d8-91a0-70225c5e3a4a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=314803578]], execName=null], jobId=800890ff961-7ff6a786-9d4d-43d8-91a0-70225c5e3a4a]] class org.apache.ignite.IgniteCheckedException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2] at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9908) at org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:438) at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1117) at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1921) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2] at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875) at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762) at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310) at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99) at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9902) ... 10 more Caused by: class org.apache.ignite.binary.BinaryInvalidTypeException: Unknown pair [platformId=0, typeId=-1409390795] at org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:696) at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1755) at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) at org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1799) at org.apache.ignite.internal.binary.BinaryReaderExImpl.readObject(BinaryReaderExImpl.java:1329) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.readBinary(GridClosureProcessor.java:1872) at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:834) ... 16 more Caused by: java.lang.ClassNotFoundException: Unknown pair [platformId=0, typeId=-1409390795] at org.apache.ignite.internal.MarshallerContextImpl.getClassName(MarshallerContextImpl.java:385) at org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerContextImpl.java:335) at org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:687) ... 22 more
update - wanted to note this is happening a fresh deployment. There are no old files or persisted anything around. All classes are either pulled from even or freshly compiled. ty.
Pavel, here is the (Scala) code (redacted)
object MyCallable {
type FooList = Array[Foo]
}
class MyCallable(cacheName: String) extends IgniteCallable[FooList] with Serializable with LazyLogging {
@IgniteInstanceResource
private var ignite: Ignite = _
override def call(): FooList = {
logger.debug("callable called.");
val fooCache = ignite.getOrCreateCache[String, Foo](cacheName)
val qry = new ScanQuery[String, Foo]()
qry.setLocal(true)
val cursor = fooCache.query(qry)
val ret = cursor.iterator().asScala.map(e => e.getValue).toArray
logger.info("load status array: {}", ret.mkString)
return ret
}
@IgniteInstanceResource
def setIgnite(ignite: Ignite): Unit = {
this.ignite = ignite
}
}