I'm currently researching the following issue, which occurs in our project under high load:
class org.apache.ignite.IgniteDeploymentException: Failed to deploy user message: java.nio.HeapByteBuffer[pos=0 lim=180 cap=180]
at org.apache.ignite.internal.util.IgniteUtils$8.apply(IgniteUtils.java:825)
at org.apache.ignite.internal.util.IgniteUtils$8.apply(IgniteUtils.java:823)
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:944)
at org.apache.ignite.internal.IgniteMessagingImpl.send0(IgniteMessagingImpl.java:106)
at org.apache.ignite.internal.IgniteMessagingImpl.send(IgniteMessagingImpl.java:82)
...
Caused by: class org.apache.ignite.internal.IgniteDeploymentCheckedException: Failed to deploy user message: java.nio.HeapByteBuffer[pos=0 lim=180 cap=180]
at org.apache.ignite.internal.managers.communication.GridIoManager.sendUserMessage(GridIoManager.java:1571)
at org.apache.ignite.internal.IgniteMessagingImpl.send0(IgniteMessagingImpl.java:103)
... 25 more
In short, the case is as follows:
- Distributed cache on three nodes, all nodes run on a single workstation (in this test);
- Workers on each node;
- Messaging between workers is done using IgniteMessaging (topic has the type of String and I've tried both byte[] and ByteBuffer as a message class);
- Client connects to the cluster and triggers some business logic, that causes cross-node messaging and scan queries. Queries and messaging are being performed concurrently.
We use peer classloading with continuous deployment mode (the problem is the same with shared mode). According to the Ignite documentation, the classes should be redeployed only if the version is changed, but it does not seem to work.
I've noticed lots of similar messages in the logs:
2017-05-05 13:31:28 INFO org.apache.ignite.logger.java.JavaLogger info Removed undeployed class: GridDeployment [ts=1493980288578, depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@38815daa, clsLdrId=36c3828db51-0d65e7d5-77bf-444d-9b8b-d18bde94ad13, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=true, usage=0]
...
2017-05-05 13:31:29 INFO org.apache.ignite.logger.java.JavaLogger info Removed undeployed class: GridDeployment [ts=1493980289125, depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@355f6680, clsLdrId=1dd3828db51-1b20df7a-a98d-45a3-8ab6-e5d229945830, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=true, usage=0]
...
This is when I use ByteBuffer as message type. In case of byte[], class B[ is being constantly re-deployed.
I've analyzed the Ignite kernel a bit, and got a suspicion that undeploy is being triggered for all classes in a classloader, when at least one class that resides in that classloader was re-deployed in some other loader.
It happens inside org.apache.ignite.spi.deployment.local.LocalDeploymentSpi#register
- At first, we get a "Map of new resources added for registered class loader" using LocalDeploymentSpi#addResource.
- Then we "Remove resources for all class loaders except {@code ignoreClsLdr}." using LocalDeploymentSpi#removeResources. Inside this method, it looks like we add all loaders that contain the old version of the new resource to a "doomed" collection.
- Finally, we iterate this collection and call onClassLoaderReleased for each element. The latter action actually causes all the classes to be undeployed (finally causing the "Removed undeployed class" messages).
I don't understand this concept. Why are there multiple classloaders? Why do we undeploy the whole classloader in such cases?
I'd be grateful, if someone could explain, how does peer classloading work in Ignite "under the hood".
P.S. I'm looking at the sources of a fresh snapshot of Ignite 2.1.0, but the behavior is the same with the standard Ignite 1.9.0.
