JBoss: Deployment fails with ClassNotFoundException even though the class is there - JaxrsScanningProcessor

Question

I have a WAR file with REST services. It deploys just fine on JBoss EAP 6.2 (corresponding to AS 7.something) in the standalone mode but it fails sometimes - but often - in the domain mode with a ClassNotFoundException for my subclass of the the JAX-RS Application class, even though it is in the war (well, it runs in standalone). Since it sometimes works, I suspect there is some concurrency issue that leads to JBoss trying to load the class before it can see it.

This is the error:

ERROR [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000001: Failed to start
 service jboss.deployment.unit.myapp.POST_MODULE: org.jboss.msc.service.StartException in
 service jboss.deployment.unit.myapp.POST_MODULE: JBAS018733: Failed to process phase POST_MODULE of deployment "myapp"
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:127)
        ...
Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: JBAS011230: Could not load JAX-RS Application class
        at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.scan(JaxrsScanningProcessor.java:218)
        at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.deploy(JaxrsScanningProcessor.java:100)
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:120)
           [jboss-as-server-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
        ... 5 more
Caused by: java.lang.ClassNotFoundException: myapp.rs.RestApplication from [Module "deployment.myapp:main" from Service Module Loader]
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:197)
           [jboss-modules.jar:1.3.0.Final-redhat-2]
        ...
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:118) [jboss-modules.jar:1.3.0.Final-redhat-2]
        at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.scan(JaxrsScanningProcessor.java:214)
        ... 7 more

The class is there:

[jboss/domain]$ unzip -l data/content/6c/0ffc675ff1c2254540b6e8caffc0d2605ed549/content | grep RestApp
     2262  02-13-14 09:05   WEB-INF/classes/myapp/rs/RestApplication.class

This is how I deployed it (my user != the user running jboss, if that makes a difference):

me$ /opt/jboss/bin/jboss-cli.sh -c --controller=0.0.0.0:49999 --user=admin --password=***
[[email protected]:49999 /] deploy /vagrant/myapp.war --all-server-groups

The deployment fails in the same way when executed via the web Admin Console. The deployment to standalone is done by copying the .war to the standalone/deployments/ directory.

Any tips?

Config info

1. JBoss EAP 6.2
2. Rest Easy 3.0.6 (replacing the default 2.x module)
3. We have no JAX-RS stuff in web.xml aside of the resteasy.servlet.mapping.prefix context param; we use the resteasy-servlet-initializer library instead of manually configuring a servlet or similar thing in there

Answer 1

It seems I have fixed the problem by removing all resteasy/jaxrs/jboss libraries from the WAR file's WEB-INF/lib/.

I have noticed the war contained many libraries it should not, such as resteasy-jaxrs-3.0.6.Final.jar, jboss-jaxr-api_1.0_spec-1.0.0.Final.jar (which, BTW, conflicts with Rest Easy 3.0.6), javax.servlet-api-3.1.0.jar etc.

So I guess the classloader did not see/find the class because of classpath hell, the server and the webapp using different "instances" of the same libraries. (No idea why it worked under JBoss standalone; but standalone has the out-of-the-box configuration while domain has been adjusted a lot).

I have also sometimes included --runtime-name=myapp in the deployment command, which seems to be wrong, I should have used --runtime-name=myapp.war (including the ending).

Answer 2

Even though the Author had found an answer for himself, I think it might be useful to post my situation in case other people come accross this issue and the authors solution doesn't work for them.

I had a similar issue where a class was also not found when run under a Redhat JBoss AS cartridge inside Openshift, even though it seemed unreasonable, but later I figured out that the war I was expecting to be deployed wasn't deployed, instead a different one was deployed.

ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/].[HelloWorld]] (http--127.10.254.129-8080-1) Allocate exception for servlet HelloWorld: java.lang.ClassNotFoundException: com.company.app.HelloWorld from [Module "deployment.ROOT.war:main" from Service Module Loader]
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)

So if there are possibilities, I would run a search of all war files which are expected to have that class, unzip them and search inside the contents to see whether they contain the needed compiled class. I really suggest trying to search for all possible wars. Also look for possible YOUR_WAR.war.deployed it is most likely going to be in the same folder as the war that is currently deployed.

In my case I had these wars:

./app-root/runtime/repo/deployments/ROOT.war
./app-root/runtime/dependencies/jbossas_custom/deployments/ROOT.war.deployed
./app-root/runtime/dependencies/jbossas_custom/deployments/ROOT.war

I was expecting the real one to be in ./app-root/runtime/repo/deployments/ROOT.war because that's where maven was creating it, but seems JBoss would later have to copy it to ./app-root/runtime/dependencies/jbossas_custom/deployments/ROOT.war in my case it didn't do that itself, so therefore my updated war was never deployed, instead it always went to deploy the old one which was already in ./app-root/runtime/dependencies/jbossas_custom/deployments/ROOT.war

Hope this helps somebody to get on the right track.