We have a situation here where more than 100 threads are stuck in a weblogic (10.3.6) managed server.
I am analysing a threaddump and find 150 threads blocked waiting on a lock in the classloader, like :
"jmsContainer-14" prio=10 tid=0x00007f11485d7000 nid=0x14c8 waiting for monitor entry [0x00007f1023590000]
java.lang.Thread.State: BLOCKED (on object monitor)
at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:48)
- waiting to lock <0x0000000782aeb288> (a weblogic.utils.classloaders.ChangeAwareClassLoader)
at ch.qos.logback.classic.spi.PackagingDataCalculator.loadClass(PackagingDataCalculator.java:207)
(10 occurences)
or
"[STUCK] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f10d002d800 nid=0x5ea5 waiting for monitor entry [0x00007f1136eeb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:48)
- waiting to lock <0x0000000782aeb288> (a weblogic.utils.classloaders.ChangeAwareClassLoader)
at javax.xml.datatype.FactoryFinder.getProviderClass(FactoryFinder.java:115)
at javax.xml.datatype.FactoryFinder.newInstance(FactoryFinder.java:181)
(137) occurences
No all stuck thread are blocked for the same reason, but we have 150 threads waiting to lock the same lock <0x0000000782aeb288>, wich I find strange because it is in the classLoader.
I don't think it is the root cause of the problem we are experiencing right now, but if someone could explain this behavious it could be of great help.
Thanks
EDIT :
Here is the blocking thread :
ExecuteThread '136' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f11484af800 nid=0x7640 runnable [0x00007f102c11b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.cs.UTF_8$Encoder.encode(UTF_8.java:632)
at java.util.zip.ZipCoder.getBytes(ZipCoder.java:86)
at java.util.zip.ZipFile.getEntry(ZipFile.java:306)
- locked <0x0000000782b2ddf8> (a java.util.jar.JarFile)
at java.util.jar.JarFile.getEntry(JarFile.java:226)
at weblogic.utils.classloaders.ZipClassFinder.getSource(ZipClassFinder.java:39)
at weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:50)
at weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:31)
at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
at weblogic.application.utils.CompositeWebAppFinder.getClassSource(CompositeWebAppFinder.java:88)
at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
at weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:25)
at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:291)
- eliminated <0x0000000782aeb288> (a weblogic.utils.classloaders.ChangeAwareClassLoader)
at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:270)
at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:64)
at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:49)
- locked <0x0000000782aeb288> (a weblogic.utils.classloaders.ChangeAwareClassLoader)
at javax.xml.datatype.FactoryFinder.getProviderClass(FactoryFinder.java:115)
at javax.xml.datatype.FactoryFinder.newInstance(FactoryFinder.java:181)
sorry, I should have posted it earlier.
Nearly all the threads are bloked in, or blocked by one of two lines of codes containing this piece of code:
DatatypeFactory.newInstance().newXMLGregorianCalendar(dateDebCre)
Apparently Weblogic is trying very hard to load the same jar again and again. Which I do not understand, because it is supposed to be stored in a cache in memory, right ?
First, we will remove this lines of code as they are completely useless (a shame it is). But again, I think understanding the problem would be better.
Thank you
