Previously 2 JBoss 6 servers (master & slave) were started with 2 different IP addresses (127.0.0.1 and 127.0.0.2) and those 2 server instances were run in domain mode. The instances used HornetQ to communicate index jobs for Hibernate Search. I'm trying to create a similar configuration using Wildfly instead of JBoss.
The master instance is run with the following configuration (fragments posted only) using port-offset = "220":
domain.xml-Master:
<server-group name="other-server-group" profile="full-ha">
<jvm name="default">
<heap size="256m" max-size="1303m"/>
<permgen max-size="256m"/>
</jvm>
<socket-binding-group ref="full-ha-sockets"/>
<deployments>
<deployment name="mc-ds.xml" runtime-name="mc-ds.xml"/>
<deployment name="abc.ear" runtime-name="abc.ear"/>
</deployments>
</server-group>
I'm trying to use Undertow to handle the connections and HornetQ for JMS with the following configuration:
<subsystem xmlns="urn:jboss:domain:messaging:2.0">
<hornetq-server>
<security-enabled>false</security-enabled>
<cluster-password>${jboss.messaging.cluster.password:root}</cluster-password>
<journal-file-size>102400</journal-file-size>
<connectors>
<http-connector name="http-connector" socket-binding="messaging">
<param key="http-upgrade-endpoint" value="http-acceptor"/>
</http-connector>
<http-connector name="http-connector-throughput" socket-binding="messaging-throughput">
<param key="http-upgrade-endpoint" value="http-acceptor-throughput"/>
<param key="batch-delay" value="50"/>
</http-connector>
<in-vm-connector name="in-vm" server-id="0"/>
</connectors>
<acceptors>
<http-acceptor http-listener="acceptors-messaging" name="http-acceptor"/>
<http-acceptor http-listener="acceptors-messaging-throughput" name="http-acceptor-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</http-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
</acceptors>
...
<subsystem xmlns="urn:jboss:domain:undertow:1.1">
<buffer-cache name="default"/>
<server name="default-server">
<ajp-listener name="ajp" socket-binding="ajp"/>
<http-listener name="default" socket-binding="http"/>
<http-listener name="remoting-connector" socket-binding="remoting"/>
<http-listener name="acceptors-messaging" socket-binding="messaging"/>
<http-listener name="acceptors-messaging-throughput" socket-binding="messaging-throughput"/>
<host name="default-host" alias="localhost">
<location name="/" handler="welcome-content"/>
<filter-ref name="server-header"/>
<filter-ref name="x-powered-by-header"/>
</host>
</server>
<servlet-container name="default">
<jsp-config/>
</servlet-container>
<handlers>
<file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
</handlers>
<filters>
<response-header name="server-header" header-name="Server" header-value="WildFly/8"/>
<response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
</filters>
</subsystem>
...
<socket-binding name="messaging" port="5445"/>
<socket-binding name="messaging-throughput" port="5455"/>
The same I did with the remoting.
<subsystem xmlns="urn:jboss:domain:remoting:2.0">
<endpoint worker="default"/>
<http-connector name="http-remoting-connector" connector-ref="remoting-connector" security-realm="ApplicationRealm"/>
</subsystem>
...
<socket-binding name="remoting" port="${jboss.http.port:4447}"/>
<socket-binding name="messaging-group" port="0" multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9876}"/>
This allows me to start the master instance and deploy the application. The instance listens on port 4667 (4447 + port offset 220).
However deploying the application on the slave instance fails as it seemingly fails to connect to the master instance. Failure only happens with Wildfly but not with JBoss.
Even though an in-vm connector is configured, too, the slave instance is supposed to connect using the network interface.
host.xml - Slave
...
<property name="hibernate.search.default.worker.jndi.java.naming.provider.url" value="remote://127.0.0.1:4667"/>
...
The slave can not load the message queue, because the connection with JNDI fails as the server.log of the slave shows. The slave runs on the same machine as the master in the local test setup.
Capturing the network traffic shows that the master acknowledges (TCP ACK) the connection to its port 4667 despite “ConnectException: Operation failed with status WAITING after 5000 MILLISECONDS” as found in the log below.
Any pointers or hints are appreciated.
Error message from the server.log - Slave:
2015-03-26 09:34:43,226 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 59) MSC000001: Failed to start service jboss.persistenceunit."abc.ear/web.war#mc": org.jboss.msc.service.StartException in service jboss.persistenceunit."abc.ear/web.war#mc": org.hibernate.search.SearchException: HSEARCH000103: Unable to initialize IndexManager AssetDerivate
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:172) [wildfly-jpa-8.1.0.Final.jar:8.1.0.Final]
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:117) [wildfly-jpa-8.1.0.Final.jar:8.1.0.Final]
at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_40]
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:474)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:182) [wildfly-jpa-8.1.0.Final.jar:8.1.0.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_40]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_40]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_40]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: org.hibernate.search.SearchException: HSEARCH000103: Unable to initialize IndexManager AssetDerivate
at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:265) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:528) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManagers(IndexManagerHolder.java:495) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.indexes.impl.IndexManagerHolder.buildEntityIndexBinding(IndexManagerHolder.java:104) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.spi.SearchFactoryBuilder.initDocumentBuilders(SearchFactoryBuilder.java:363) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.spi.SearchFactoryBuilder.buildNewSearchFactory(SearchFactoryBuilder.java:219) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.spi.SearchFactoryBuilder.buildSearchFactory(SearchFactoryBuilder.java:143) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.hcore.impl.HibernateSearchSessionFactoryObserver.sessionFactoryCreated(HibernateSearchSessionFactoryObserver.java:74) [hibernate-search-orm-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.internal.SessionFactoryObserverChain.sessionFactoryCreated(SessionFactoryObserverChain.java:52) [hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:588) [hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1857) [hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:850) [hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:843) [hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.withTccl(ClassLoaderServiceImpl.java:397) [hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:842) [hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
at org.jboss.as.jpa.hibernate4.TwoPhaseBootstrapImpl.build(TwoPhaseBootstrapImpl.java:44) [jipijapa-hibernate4-3-1.0.1.Final.jar:]
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:154) [wildfly-jpa-8.1.0.Final.jar:8.1.0.Final]
... 8 more
Caused by: org.hibernate.search.SearchException: HSEARCH000164: Unable to lookup Search queue 'queue/hibernatesearch' and connection factory 'jms/RemoteConnectionFactory' for index 'AssetDerivate'
at org.hibernate.search.backend.impl.jms.JndiJMSBackendQueueProcessor.initializeJMSQueueConnectionFactory(JndiJMSBackendQueueProcessor.java:56) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.backend.impl.jms.JmsBackendQueueProcessor.initialize(JmsBackendQueueProcessor.java:77) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.backend.BackendFactory.createBackend(BackendFactory.java:85) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.backend.BackendFactory.createBackend(BackendFactory.java:53) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.createBackend(DirectoryBasedIndexManager.java:204) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:102) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:261) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
... 24 more
Caused by: javax.naming.CommunicationException: Failed to connect to any server. Servers tried: [remote://127.0.0.1:4667 (Operation failed with status WAITING after 5000 MILLISECONDS)] [Root exception is java.net.ConnectException: Operation failed with status WAITING after 5000 MILLISECONDS]
at org.jboss.naming.remote.client.HaRemoteNamingStore.failOverSequence(HaRemoteNamingStore.java:240) [jboss-remote-naming-2.0.1.Final.jar:2.0.1.Final]
at org.jboss.naming.remote.client.HaRemoteNamingStore.namingStore(HaRemoteNamingStore.java:149) [jboss-remote-naming-2.0.1.Final.jar:2.0.1.Final]
at org.jboss.naming.remote.client.HaRemoteNamingStore.namingOperation(HaRemoteNamingStore.java:130) [jboss-remote-naming-2.0.1.Final.jar:2.0.1.Final]
at org.jboss.naming.remote.client.HaRemoteNamingStore.lookup(HaRemoteNamingStore.java:272) [jboss-remote-naming-2.0.1.Final.jar:2.0.1.Final]
at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:87) [jboss-remote-naming-2.0.1.Final.jar:2.0.1.Final]
at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:129) [jboss-remote-naming-2.0.1.Final.jar:2.0.1.Final]
at javax.naming.InitialContext.lookup(InitialContext.java:417) [rt.jar:1.8.0_40]
at javax.naming.InitialContext.lookup(InitialContext.java:417) [rt.jar:1.8.0_40]
at org.hibernate.search.backend.impl.jms.JndiJMSBackendQueueProcessor.initializeJMSQueueConnectionFactory(JndiJMSBackendQueueProcessor.java:53) [hibernate-search-engine-4.5.1.Final.jar:4.5.1.Final]
... 30 more
Caused by: java.net.ConnectException: Operation failed with status WAITING after 5000 MILLISECONDS
at org.jboss.naming.remote.protocol.IoFutureHelper.get(IoFutureHelper.java:97) [jboss-remote-naming-2.0.1.Final.jar:2.0.1.Final]
at org.jboss.naming.remote.client.HaRemoteNamingStore.failOverSequence(HaRemoteNamingStore.java:198) [jboss-remote-naming-2.0.1.Final.jar:2.0.1.Final]
... 38 more
ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([("deployment" => "abc.ear")]) - failure description: {"JBAS014671: Failed services" => {"jboss.persistenceunit.\"abc.ear/web.war#mc\"" => "org.jboss.msc.service.StartException in service jboss.persistenceunit.\"abc.ear/web.war#mc\": org.hibernate.search.SearchException: HSEARCH000103: Unable to initialize IndexManager AssetDerivate
Caused by: org.hibernate.search.SearchException: HSEARCH000103: Unable to initialize IndexManager AssetDerivate
Caused by: org.hibernate.search.SearchException: HSEARCH000164: Unable to lookup Search queue 'queue/hibernatesearch' and connection factory 'jms/RemoteConnectionFactory' for index 'AssetDerivate'
Caused by: javax.naming.CommunicationException: Failed to connect to any server. Servers tried: [remote://127.0.0.1:4667 (Operation failed with status WAITING after 5000 MILLISECONDS)] [Root exception is java.net.ConnectException: Operation failed with status WAITING after 5000 MILLISECONDS]
Caused by: java.net.ConnectException: Operation failed with status WAITING after 5000 MILLISECONDS"}}
