ClassNotFoundException in tomcat for an inner class in servlet code

Question

We have a web-application, instance of which get deployed multiple times under the same tomcat (v6.0.14) instance. Suppose the war file is named "app.war", then we deploy "app1.war", "app2.war" and so on.

All the application instances (~20) get deployed correctly. When the servlets on these applications are accessed, sometimes they fail with NoClassDefFoundError caused by ClassNotFoundException in tomcat's WebAppClassLoader. Following is one example -

java.lang.NoClassDefFoundError: com/xxx/APISocketServer$ClientRequestHandler
    at com.xxx.APISocketServer$APISocketServerThread.run(APISocketServer.java:143)
Caused by: java.lang.ClassNotFoundException: com.xxx.APISocketServer$ClientRequestHandler
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1358)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
    ... 1 more

The offending classes are part of application code and are not part of any library.

As some of the application instances run correctly, this cannot be a case of classpath mis-configuration. One thing to note is that there are only a couple of classes that fail to load. The classes do not have any static initializer blocks, which could cause failure of class initialization. I have checked tomcat logs (catalina.out, localhost.log) and have not found any class initialization errors.

How should I proceed to debug this problem?

Post the stack trace showing both errors. Without that we're depending on your interpretation of the problem. — Jim Garrison
Is the Java version correct? Quite often these errors occur with corrupt or incorrect JDK versions. Try setting JAVA_HOME to another JDK. — Rhys
Tomcat is running under JRE1.6U22 and JRE installation is correct and free of corruption. I've verified md5sums of files in java directory with another system. Max heap size is 4g and max perm size is 1g. System has more than enough RAM (16g) to avoid any memory constraints. — Lalit Mishra

Lalit Mishra Lalit Mishra · Accepted Answer · 2013-03-05T17:02:30

On further investigation, limit of open file descriptors (1024) was found to be the culprit. Each application instance would require own copies of Jar files opened by its class loader (WebappClassLoader). The class loader closes the files if a class from a Jar file is not accessed for 90 seconds. If several application instances are accessed within a short duration, the openJARs function would fail resulting in the class not getting loaded.

The error would have been found much earlier if WebappClassLoader had logged the failure in opening JAR file at higher log level. The error is logged at debug level. Enabling debug level for WebappClassLoader, helped in figuring out the problem.

ClassNotFoundException in tomcat for an inner class in servlet code

1 Answers