currently I'm working in a project and I have an issue implementing ELB as a load balancer and Identity Server as an Identity and PDP provider in a cluster.
I would appreciate so much any help you can give me with the following implementation:
The scenario
Two ELB in fail-over mode with a virtual IP virtual and KeepAlive, each node in separated machines
One ESB cluster ESB with one manager node and two workers nodes; these nodes are in other machines separated from the ELB nodes.
- One IS cluster with two nodes; these ones are in different machines separated from ELB and ESB machines.
- I have Open LDAP as User Store
- SVN as an artifact repository for syncing using Hazelcast
- All the nodes have a shared UM DB and a shared registry
- I have applied all the patches available for kernel 4.2.0 to the nodes of ELB, IS and ESB (patch 0010 inclusive)
The ELB nodes start correctly, the same for the ESB nodes which join the cluster without any problems; I can verify this in the ELB and ESB wso2carbon.log; also I validated the load balancing when I send requests to the ESB worker nodes from the ELB.
But, when I start the first IS node I get the following messages in wso2carbon.log of IS
TID: [0] [IS] [2015-05-18 20:09:30,178] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Cluster domain: example.wso2is.domain {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [0] [IS] [2015-05-18 20:09:30,179] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Using wka based membership management scheme {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [0] [IS] [2015-05-18 20:09:30,180] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:IP_Virtual_ELB, Remote Host:null, Port: 4050, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils}
TID: [0] [IS] [2015-05-18 20:09:35,468] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 5286ms {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [0] [IS] [2015-05-18 20:09:35,546] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [4c91357f-b563-49a2-b71b-887b3109642a] - Host:IP_Nodo1_Identity, Remote Host:null, Port: 4301, HTTP:9763, HTTPS:9443, Domain: example.wso2is.domain, Sub-domain:mgt, Active:true {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [0] [IS] [2015-05-18 20:09:35,558] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:IP_Nodo1_Identity, Remote Host:null, Port: 4301, HTTP:9763, HTTPS:9443, Domain: interacciones.wso2is.domain, Sub-domain:mgt, Active:true {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils}
TID: [0] [IS] [2015-05-18 20:09:35,650] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Cluster initialization completed {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [0] [IS] [2015-05-18 20:09:35,652] INFO {org.apache.tomcat.util.net.NioSelectorPool} - may 18, 2015 8:09:35 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector INFORMACIÓN: Using a shared selector for servlet write/read {org.apache.tomcat.util.net.NioSelectorPool}
TID: [0] [IS] [2015-05-18 20:09:35,653] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Elected this member [4c91357f-b563-49a2-b71b-887b3109642a] as the Coordinator for the cluster [example.wso2is.domain] {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
And the wso2carbon.log of ELB does not have any messages referring to this IS node that is just starting.
The services defined in loadbalancer.conf of ELB are as follows:
esb {
domains {
example.wso2esb.domain {
tenant_range *;
group_mgt_port 4000;
worker {
hosts wso2esb.example.com;
}
}
}
}
identity {
domains {
example.wso2is.domain {
tenant_range *;
group_mgt_port 4050;
mgt{
hosts wso2is.example.com;
}
}
}
}
If I try to access from a client PC the IS console using the cirtual IP of the ELB, I get the following message in the ELB log:
TID: [0] [ELB] [2015-05-19 13:13:31,705] ERROR {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - No application members available {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
the content of /etc/hosts is:
Client:
ELB_Virtual_IP wso2is.example.com
ELB_Virtual_IP wso2esb.example.com
ESB_Manager_IP mgt.wso2esb.example.com
ELB nodes:
ELB_Virtual_IP wso2is.example.com
ELB_Virtual_IP wso2esb.example.com
ESB_Manager_IP mgt.wso2esb.example.com
IS Node:
ELB_Virtual_IP wso2is.example.com
ESB nodes:
ELB_Virtual_IP wso2is.example.com
ELB_Virtual_IP wso2esb.example.com
ESB_Manager_IP mgt.wso2esb.example.com
axis2.xml of IS node (clustering section)
<parameter name="AvoidInitiation">true</parameter>
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">example.wso2is.domain</parameter>
<parameter name="mcastPort">45564</parameter>
<parameter name="mcastTTL">100</parameter>
<parameter name="mcastTimeout">60</parameter>
<parameter name="localMemberHost">IS_NODE_IP</parameter>
<parameter name="localMemberPort">4301</parameter>
<parameter name="properties">
<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
<property name="subDomain" value="mgt"/>
</parameter>
<members>
<member>
<hostName>ELB_Virtual_IP</hostName>
<port>4050</port>
</member>
</members>
<groupManagement enable="false">
<applicationDomain name="interacciones.wso2is.domain"
description="IS group"
agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
subDomain="mgt"
port="4301"/>
</groupManagement>
</clustering>
carbon.xml of IS Node
<HostName>wso2is.example.com</HostName>
<MgtHostName>wso2is.example.com</MgtHostName>
