I am trying to build a jbpm6.5 cluster with mysql as db. I m using zookeeper-3.4.10 helix-core-0.6.8
I am creating setup by following steps
./bin/helix-admin.sh --zkSvr localhost:2181,localhost:2182,localhost:2183 --addCluster jbpm-domain-cluster
/bin/helix-admin.sh --zkSvr localhost:2181,localhost:2182,localhost:2183 --addNode jbpm-domain-cluster server-one:12345
./bin/helix-admin.sh --zkSvr localhost:2181,localhost:2182,localhost:2183 --addNode jbpm-domain-cluster server-two:12346
./bin/helix-admin.sh --zkSvr localhost:2181,localhost:2182,localhost:2183 --addResource jbpm-domain-cluster vfs-repo-domain 1 LeaderStandby AUTO_REBALANCE
./bin/run-helix-controller.sh --zkSvr localhost:2181,localhost:2182,localhost:2183 --cluster jbpm-domain-cluster 2>&1 > ./controller.log &
node-one start ./bin/standalone.sh -c standalone-full-ha.xml -b 10.14.24.161
node-two start ./bin/standalone.sh -Djboss.socket.binding.port-offset=100 -c standalone-full.xml -b 10.14.24.161
I am using following node configuration node one
<system-properties>
<property name="org.kie.demo" value="false"/>
<property name="org.kie.example" value="false"/>
<property name="org.kie.server.persistence.ds" value="java:jboss/datasources/jbpmDS"/>
<property name="org.kie.server.persistence.dialect" value="org.hibernate.dialect.MySQLInnoDBDialect"/>
<property name="jboss.node.name" value="server-one"/>
<property name="org.uberfire.nio.git.dir" value="/home/rupesh/jbpm6.5Cluster/clustering/server-one"/>
<property name="org.uberfire.cluster.id" value="jbpm-domain-cluster"/>
<property name="org.uberfire.cluster.zk" value="localhost:2181,localhost:2182,localhost:2183"/>
<property name="org.uberfire.cluster.local.id" value="server-one_12345"/>
<property name="org.uberfire.cluster.vfs.lock" value="vfs-repo-domain"/>
<property name="org.uberfire.cluster.autostart" value="false"/>
<property name="org.uberfire.nio.git.daemon.port" value="9418"/>
<property name="org.uberfire.metadata.index.dir" value="/home/rupesh/jbpm6.5Cluster/clustering/server-one"/>
<property name="org.uberfire.nio.git.ssh.cert.dir" value="/home/rupesh/jbpm6.5Cluster/clustering/server-one"/>
<property name="org.quartz.properties" value="/home/rupesh/jbpm6.5Cluster/clustering/quartz/quartz-definition-mysql.properties"/>
</system-properties>
node two
<system-properties>
<property name="org.kie.demo" value="false"/>
<property name="org.kie.example" value="false"/>
<property name="jboss.node.name" value="server-two" />
<property name="org.kie.server.persistence.ds" value="java:jboss/datasources/jbpmDS" />
<property name="org.kie.server.persistence.dialect" value="org.hibernate.dialect.MySQLInnoDBDialect" />
<property name="org.uberfire.nio.git.dir" value="/home/rupesh/jbpm6.5Cluster/clustering/server-two"/>
<property name="org.uberfire.cluster.id" value="jbpm-domain-cluster"/>
<property name="org.uberfire.cluster.zk" value="localhost:2181,localhost:2182,localhost:2183"/>
<property name="org.uberfire.cluster.local.id" value="server-two_12346"/>
<property name="org.uberfire.cluster.vfs.lock" value="vfs-repo-domain" />
<property name="org.uberfire.cluster.autostart" value="false"/>
<property name="org.uberfire.nio.git.daemon.port" value="9419" />
<property name="org.uberfire.metadata.index.dir" value="/home/rupesh/jbpm6.5Cluster/clustering/server-two" />
<property name="org.uberfire.nio.git.ssh.cert.dir" value="/home/rupesh/jbpm6.5Cluster/clustering/server-two"/>
<property name="org.quartz.properties" value="/home/rupesh/jbpm6.5Cluster/clustering/quartz/quartz-definition-mysql.properties" />
</system-properties>
Node one starts properly, but while starting 2nd node it get stuck and jbpm-console never get deployed. Following is the last log that get printed in 2nd node
2017-09-19 13:31:16,649 INFO [org.apache.helix.messaging.handling.HelixTaskExecutor] (ZkClient-EventThread-432-localhost:2181,localhost:2182,localhost:2183) Dropping NO-OP message. mid: 7ffff8bb-6b3a-467d-a8c4-49140f1824a6, from: server-two_12346 2017-09-19 13:31:16,653 INFO [org.apache.helix.manager.zk.CallbackHandler] (ZkClient-EventThread-432-localhost:2181,localhost:2182,localhost:2183) 432 END:INVOKE /jbpm-domain-cluster/INSTANCES/server-two_12346/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor Took: 7ms 2017-09-19 13:31:16,653 WARN [org.apache.helix.manager.zk.CallbackHandler] (MSC service thread 1-3) Skip processing callbacks for listener: org.apache.helix.messaging.handling.HelixTaskExecutor@79bafb4f, path: /jbpm-domain-cluster/INSTANCES/server-two_12346/MESSAGES, expected types: [CALLBACK, FINALIZE] but was INIT 2017-09-19 13:31:16,653 INFO [org.apache.helix.manager.zk.ZKHelixManager] (MSC service thread 1-3) init handler: /jbpm-domain-cluster/INSTANCES/server-two_12346/MESSAGES, org.apache.helix.messaging.handling.HelixTaskExecutor@79bafb4f 2017-09-19 13:31:16,657 INFO [org.apache.helix.manager.zk.CallbackHandler] (ZkClient-EventThread-432-localhost:2181,localhost:2182,localhost:2183) 432 START:INVOKE /jbpm-domain-cluster/INSTANCES/server-two_12346/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor 2017-09-19 13:31:16,658 INFO [org.apache.helix.manager.zk.CallbackHandler] (ZkClient-EventThread-432-localhost:2181,localhost:2182,localhost:2183) server-two_12346 subscribes child-change. path: /jbpm-domain-cluster/INSTANCES/server-two_12346/MESSAGES, listener: org.apache.helix.messaging.handling.HelixTaskExecutor@79bafb4f 2017-09-19 13:31:16,661 INFO [org.apache.helix.messaging.handling.HelixTaskExecutor] (ZkClient-EventThread-432-localhost:2181,localhost:2182,localhost:2183) No Messages to process 2017-09-19 13:31:16,661 INFO [org.apache.helix.manager.zk.CallbackHandler] (ZkClient-EventThread-432-localhost:2181,localhost:2182,localhost:2183) 432 END:INVOKE /jbpm-domain-cluster/INSTANCES/server-two_12346/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor Took: 4ms 2017-09-19 13:35:51,838 ERROR [org.jboss.as.controller.management-operation] (management task-7) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[("deployment" => "jbpm-console.war")]' 2017-09-19 13:35:51,841 ERROR [org.jboss.as.server] (management task-7) WFLYSRV0021: Deploy of deployment "jbpm-console.war" was rolled back with the following failure message: "WFLYCTL0344: Operation timed out awaiting service container stability" 2017-09-19 13:35:51,852 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = jbpm-console.war_org.h2.Driver_1_3 2017-09-19 13:35:51,854 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0019: Stopped Driver service with driver-name = jbpm-console.war_com.mysql.fabric.jdbc.FabricMySQLDriver_5_1 2017-09-19 13:35:51,864 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = jbpm-console.war_com.mysql.jdbc.Driver_5_1 2017-09-19 13:35:51,866 INFO [org.jboss.as.webservices] (MSC service thread 1-2) WFLYWS0004: Stopping service jboss.ws.endpoint."jbpm-console.war".CommandWebService 2017-09-19 13:35:51,870 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 78) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/queue/KIE.SESSION.ALL 2017-09-19 13:35:51,873 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 78) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/exported/jms/queue/KIE.SESSION 2017-09-19 13:35:51,874 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 80) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/queue/KIE.AUDIT.ALL 2017-09-19 13:35:51,874 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 80) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/exported/jms/queue/KIE.AUDIT 2017-09-19 13:35:51,874 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 81) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/queue/KIE.SIGNAL 2017-09-19 13:35:51,874 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 81) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/exported/jms/queue/KIE.SIGNAL 2017-09-19 13:35:51,874 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 82) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/exported/jms/queue/KIE.RESPONSE 2017-09-19 13:35:51,875 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 82) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/queue/KIE.RESPONSE.ALL 2017-09-19 13:35:51,875 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 79) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/exported/jms/queue/KIE.EXECUTOR 2017-09-19 13:35:51,875 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 79) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/queue/KIE.EXECUTOR 2017-09-19 13:35:51,875 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 83) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/queue/KIE.TASK.ALL 2017-09-19 13:35:51,875 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 83) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/exported/jms/queue/KIE.TASK 2017-09-19 13:35:56,841 ERROR [org.jboss.as.controller.management-operation] (management task-7) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@152fff29 for operation {"operation" => "add","address" => [("deployment" => "jbpm-console.war")],"runtime-name" => "jbpm-console.war","enabled" => true,"content" => [{"input-stream-index" => 0}],"operation-headers" => {"access-mechanism" => "HTTP"},"persistent" => true,"owner" => undefined} at address [("deployment" => "jbpm-console.war")] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511) at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369) at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328) at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1301) at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185) at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767) at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344) at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392) at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:204) at org.jboss.as.domain.http.server.DomainApiGenericOperationHandler.handleRequest(DomainApiGenericOperationHandler.java:179) at io.undertow.server.handlers.encoding.EncodingHandler.handleRequest(EncodingHandler.java:72) at org.jboss.as.domain.http.server.security.SubjectDoAsHandler$1.run(SubjectDoAsHandler.java:72) at org.jboss.as.domain.http.server.security.SubjectDoAsHandler$1.run(SubjectDoAsHandler.java:68) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:92) at org.jboss.as.domain.http.server.security.SubjectDoAsHandler.handleRequest(SubjectDoAsHandler.java:68) at org.jboss.as.domain.http.server.security.SubjectDoAsHandler.handleRequest(SubjectDoAsHandler.java:63) at io.undertow.server.handlers.BlockingHandler.handleRequest(BlockingHandler.java:56) at org.jboss.as.domain.http.server.DomainApiCheckHandler.handleRequest(DomainApiCheckHandler.java:87) at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:52) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-09-19 13:36:01,842 ERROR [org.jboss.as.controller.management-operation] (management task-7) WFLYCTL0349: Timeout after [5] seconds waiting for service container stability while finalizing an operation. Process must be restarted. Step that first updated the service container was 'add' at address '[("deployment" => "jbpm-console.war")]'
Please help. Thanks
