Huge lag for WSO2 SVN Synchronizer to sync with manager updates on cluster

0
votes

I'm running a test environment on WSO2 APIM 1.10.0 on VM on my Windows PC. It is configured to use MySQL server (MiraDB which I run on my PC as well). Everything was working ok.

Recently I wanted to try a WSO2 cluster environment by setting up 3 VM's on my PC:

  • The first one is running the publisher, store, KM, etc (I'm using offset 1, so ports are 9444, 9764, etc..)
  • The other two each run a gateway worker ("guest" port 8243, mapped to "host" ports 8243 and 8943).

I'm also running VisualSVN server to synchronize between them all.

On the manager node, SVN synchronizer is configured as:

   <DeploymentSynchronizer>
    <Enabled>true</Enabled>
    <AutoCommit>true</AutoCommit>
    <AutoCheckout>true</AutoCheckout>
    <RepositoryType>svn</RepositoryType>
    <SvnUrl>https://10.0.2.2/svn/apigw/</SvnUrl>
    <SvnUser>...</SvnUser>
    <SvnPassword>...</SvnPassword>
    <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>

And on the worker nodes:

    <DeploymentSynchronizer>
    <Enabled>true</Enabled>
    <AutoCommit>false</AutoCommit>
    <AutoCheckout>true</AutoCheckout>
    <RepositoryType>svn</RepositoryType>
    <SvnUrl>https://10.0.2.2/svn/apigw/</SvnUrl>
    <SvnUser>...</SvnUser>
    <SvnPassword>...</SvnPassword>
    <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>

AXIS2 is configured for clustering with

  • Manager node on port 4500
  • Worker nodes on port 4100 and 4200

I checked using telnet and all ports are accessible from all nodes.

Changes to API's on the manager are submitted correctly to the SVN. I checked on both Visual SVN and on a command line SVN client. For example, after adding API ofer3, revision #11 created, and was seen by the command line SVN tool:

> svn.exe revert .
> svn.exe update  . -r HEAD --depth=infinity
Updating '.':
U    -1234\synapse-configs\default\api\admin--ofer3_v1.0.0.xml
Updated to revision 11.

But it takes about 10 minutes before changes done on the manager nodes are populated to the worker.

For example, adding ofer2 API at the manager on 16:29

TID: [-1234] [] [2017-03-07 16:29:01,156]  INFO {org.apache.synapse.rest.API} -  Initializing API: admin--ofer2:v1.0.0
TID: [-1234] [] [2017-03-07 16:29:16,104]  INFO {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} -  Sent [SynchronizeRepositoryRequest{tenantId=-1234, tenantDomain='carbon.super', messageId=64959660-b2e6-4293-ad9c-3b0d68229976}]

Arrived to the client on 16:34, 5 minutes later:

TID: [-1234] [] [2017-03-07 16:34:14,134]  INFO {org.apache.synapse.rest.API} -  Initializing API: admin--ofer2:v1.0.0
TID: [-1234] [] [2017-03-07 16:34:14,134]  INFO {org.apache.synapse.deployers.APIDeployer} -  API named 'admin--ofer2:v1.0.0' has been deployed from file : /AppMount/wso2worker-1.10.0/repository/deployment/server/synapse-configs/default/api/admin--ofer2_v1.0.0.xml

And many times it took even more (9-10 minutes).

I turned on synchronizing debugger on the worker, expecting to see it trying to sync with SVN repository every few seconds, but only saw it trying to do so every few minutes.

Also tried with:

<SynchronizationPeriod>1</SynchronizationPeriod>

But it did not change anything.

As for log messages

On the worker log, I see:

TID: [-1234] [] [2017-03-07 15:07:31,431] ERROR {org.apache.catalina.loader.WebappClassLoa
der} -  The web application [/api/am/publisher/v0.9] appears to have started a TimerThread
 named [Timer-8] via the java.util.Timer API but has failed to stop it. To prevent a memor
y leak, the timer (and hence the associated thread) has been forcibly canceled.

But /api/am/publisher/0.9 is publisher's REST, which is not related.

Nothing else in the log seems interesting.

Final note: Changes to tenants are not populated at all. I have tenant #1 on the manager, and I do see it on the SVN repository, but on the worker a directory /AppMount/wso2/repository/tenants is empty. Only changes to super-carbon [-1234] are populated. Not sure if that's the same issue, or something else.

Any ideas will be highly appreciated.

FOLLOW UP #1, based on input from Pubci

  • Time is synchronized b/w all three nodes
  • Domain is identical in all three nodes (I left the default value)

axis2.xml of manager (10.0.2.2 is the address of host of the VM, so it serves as the "bridge" from one VM to the other)

<parameter name="domain">wso2.am.domain</parameter>
<parameter name="membershipScheme">wka</parameter>
<parameter name="localMemberPort">4500</parameter>
<members>
    <member><hostName>127.0.0.1</hostName><port>4500</port></member>
    <member><hostName>10.0.2.2</hostName><port>4100</port></member>
    <member><hostName>10.0.2.2</hostName><port>4200</port></member>
</members>

axis2.xml of worker node 1:

<parameter name="domain">wso2.am.domain</parameter>
<parameter name="membershipScheme">wka</parameter>
<parameter name="localMemberPort">4500</parameter>
<members>
    <member><hostName>10.0.2.2</hostName><port>4500</port></member>
    <member><hostName>127.0.0.1</hostName><port>4100</port></member>
    <member><hostName>10.0.2.2</hostName><port>4200</port></member>
</members>

When worker is coming up, it lists the following members:

TID: [-1234] [] [2017-03-08 09:40:39,450]  INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} -  Added member: Host:10.0.2.2, Remote Host:null, Port: 4500, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
TID: [-1234] [] [2017-03-08 09:40:39,450]  INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} -  Added member: Host:127.0.0.1, Remote Host:null, Port: 4100, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
TID: [-1234] [] [2017-03-08 09:40:39,451]  INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} -  Added member: Host:10.0.2.2, Remote Host:null, Port: 4200, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true

Note the "Domain: null" in the log. Is this ok?

When worker is coming up, it synchronizes correctly with the SVN repository:

TID: [-1234] [] [2017-03-08 09:40:51,184] DEBUG {org.wso2.carbon.deployment.synchronizer.subversion.SVNNotifyListener} -  revert /AppMount/wso2/repository/deployment/server
TID: [-1234] [] [2017-03-08 09:40:58,139] DEBUG {org.wso2.carbon.deployment.synchronizer.subversion.SVNNotifyListener} -  update /AppMount/wso2/repository/deployment/server -r HEAD --depth=infinity
TID: [-1234] [] [2017-03-08 09:40:59,766] DEBUG {org.wso2.carbon.deployment.synchronizer.subversion.SVNNotifyListener} -  notify.at
TID: [-1234] [] [2017-03-08 09:41:00,103] DEBUG {org.wso2.carbon.deployment.synchronizer.subversion.SVNBasedArtifactRepository} -  files were updated to revision number: 15 using SVN Kit

From then on, every 15 seconds the carbon scheduler tasks says it runs SVN sync:

TID: [-1234] [] [2017-03-08 09:41:45,213] DEBUG {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} -  
    Running deployment synchronizer update... tenant : carbon.super

But the SVN synchronizer does not seem to update the files in synapse-config under server/deployment.

You mentioned this is because the message from the manager does not reach at the worker.

I do see the manager send a message:

TID: [-1234] [] [2017-03-08 08:49:48,121]  
    INFO {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} -  
    Sent [SynchronizeRepositoryRequest{tenantId=-1234, tenantDomain='carbon.super', 
    messageId=a99ff1fc-58d8-44dd-8804-491216ae1a7c}]

Which debug should I enable to see if the message arrives at the worker?

2
wso2wso2-amwso2carbon

2 Answers

1
votes

For the troubleshooting, you can check the following.

  1. Clustering configurations in Axis2.xml - As you are running multiple profiles in a single node you need to cluster all the 3 nodes as one cluster. So the domain name should be same in all the 3 nodes.
  2. Time should be synched between all the 3 nodes.

Once you publish an API, cluster message is sent to the worker nodes. Then only worker nodes will get the update from the SVN.

Regarding the error message you got in the manager node, please check the AuthManager configuration in api-manager.xml. Looks like you have set the value as admin/services. That value should be key manager node hostname. In your case, it should be the hostname of the manager node.

0
votes

Thanks to the input from Pubci I found the issue.

a) In axis2.xml of both manager and workers, the localMemberHost must be 10.0.2.2 (this is the gateway from the VM to the other hosts) and not 127.0.0.1:

<parameter name="localMemberHost">10.0.2.2</parameter>

b) Also in axis2.xml I enabled groupManagement, which was disabled. In the manager node:

    <groupManagement enable="true">
        <applicationDomain name="wso2.apim.domain"
                           description="APIM group"
                           agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
                           subDomain="mgt"
                           port="2233"/>
    </groupManagement>

In the worker node:

    <groupManagement enable="true">
        <applicationDomain name="wso2.apim.domain"
                           description="APIM group"
                           agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
                           subDomain="worker"
                           port="2233"/>
    </groupManagement>

(I'm using port 2233 instead of 2222, which is the default, as port 2222 is used for other purposes in my cluster).

Now in manager I see:

 INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -  
Member joined [6bf6ae47-bea4-4bc4-beec-9140a626781b]: /10.0.2.2:4200

And in the worker, following API changes I do see the message coming in, also for tenants other than super.carbon:

INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusterMessageListener} -
  Received ClusteringMessage: SynchronizeRepositoryRequest{tenantId=1, tenantDomain='0000s7.com', messageId=a573eeef-46d7-4a2b-bfc9-362296bb60d4}

Tips for anyone having issues with SVN on cluster:

  1. Make sure the list of members that hazelcast displays when WSO2 is coming up is correct.
  2. Make sure you see Member Joins messages at the manager log when the worker is coming up.
  3. Make sure you see "Message Received" at the worker after changes at the manager.

Debugging options to help you out:

log4j.logger.org.wso2.carbon.core.deployment=DEBUG
log4j.logger.org.wso2.carbon.deployment.synchronizer=DEBUG