No APIs published on the store in clustered mode - wso2 api manager 2.0.0

Question

I am deployed Clustered API Manager 2.0.0 installation with five VMs:

VM1 shared key manager + analytics node:

Dedicated key manager running with /var/wso2am/bin/wso2server.sh -Dprofile=api-key-manager
wso2am analytics running with default port offset (+1) by /var/wso2am-analytics/bin/wso2server.sh

VM2

dedicated gateway node running with default offset (+0) by /var/wso2am/bin/wso2server.sh -Dprofile=gateway-manager

VM3

dedicated store node running with default offset (+0) by /var/wso2am/var/wso2am/bin/wso2server.sh -Dprofile=api-store

VM4

dedicated traffic manager node running with default offset (+0) by /var/wso2am/bin/wso2server.sh -Dprofile=traffic-manager

VM5

dedicated publisher node running with default offset (+0) by /var/wso2am/bin/wso2server.sh -Dprofile=api-publisher

I completed all steps in Clustering API Manager 2.0.0

I setup two cluster nodes with VM3 and VM5, and I see /_system/governance/apimgt/applicationdata registry on both store and publisher nodes. I also can browse APIs created in publisher node on store carbon admin interface (metadata->list>APIs)

But there no APIs appear in the dedicated store frond end on VM3 ("VM3/store").

In the same time if I will open "VM5/store" page on publisher node, I am can to see published API just fine

What should I check?

UPD:

The APIs is pushed to the store node with wrong Lifecycle Status "created" that is not updated accordingly. I tried to change Lifecycle Status manually on the store node and API is published on store mode just fine.

So seems issue is with Lifecycle Status live update to the store node. How can I fix it?

UPD2:

Here is my axis2.xml config on store (ip is x.x.x.15)

   <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
                enable="true">
   <parameter name="membershipScheme">wka</parameter>
   <parameter name="domain">wso2.carbon.domain</parameter>
   <parameter name="localMemberHost">x.x.x.15</parameter>
   <parameter name="localMemberPort">4000</parameter>
        <members>
            <member>
                <hostName>x.x.x.17</hostName>
                <port>4000</port>
            </member>
            <member>
                <hostName>x.x.x.15</hostName>
                <port>4000</port>
            </member>            
        </members>

`

Here is my axis2.xml config on publisher (ip is x.x.x.17)

    <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
                enable="true">
   <parameter name="membershipScheme">wka</parameter>
   <parameter name="domain">wso2.carbon.domain</parameter>
   <parameter name="localMemberHost">x.x.x.17</parameter>
   <parameter name="localMemberPort">4000</parameter>

        <members>
            <member>
                <hostName>x.x.x.15</hostName>
                <port>4000</port>
            </member>
            <member>
                <hostName>x.x.x.17</hostName>
                <port>4000</port>
            </member>            
        </members>

messages in store log file

   Cluster domain: wso2.carbon.domain
   Loading hazelcast configuration from axis2 clustering configuration
   Using wka based membership management scheme
   Added member: Host:x.x.x.17, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
   Added member: Host:x.x.x.15, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
   Hazelcast initialized in 9167ms
   Local member: [3c9c1b66-9ad9-4e4f-8d7e-de2864e8d56d] - Host:x.x.x.15, Remote Host:null, Port: 4000, HTTP:8280, HTTPS:8243, Domain: wso2.carbon.domain, Sub-domain:worker, Active:true
   Cluster initialization completed
   Task service starting in CLUSTERED mode...
   Successfully Initialized Eventing on Registry

messages in publisher log file

Cluster domain: wso2.carbon.domain
Loading hazelcast configuration from axis2 clustering configuration
Using wka based membership management scheme
Added member: Host:x.x.x.15, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
Added member: Host:x.x.x.17, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
Hazelcast initialized in 10380ms
Local member: [094f544b-1cfe-41b1-9a5f-0664aefc1ba5] - Host:x.x.x.17, Remote Host:null, Port: 4000, HTTP:8280, HTTPS:8243, Domain: wso2.carbon.domain, Sub-domain:worker, Active:true
Cluster initialization completed
Error in instantiating class : org.wso2.carbon.apimgt.usage.publisher.APIMgtThrottleUsageHandler
Sequence Deployment from the file : /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/_throttle_out_handler_.xml : Failed.
Deployment of the Synapse Artifact from file : /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/_throttle_out_handler_.xml : Failed!
The file has been backed up into : NO_BACKUP_ON_WORKER.INFO
Deployment of synapse artifact failed. Error reading /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/_throttle_out_handler_.xml : Sequence Deployment from the file : /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/_throttle_out_handler_.xml : Failed.
org.apache.axis2.deployment.DeploymentException: Sequence Deployment from the file : /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/_throttle_out_handler_.xml : Failed.
Error in instantiating class : org.wso2.carbon.apimgt.usage.publisher.APIMgtFaultHandler
Sequence Deployment from the file : /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/fault.xml : Failed.
Deployment of the Synapse Artifact from file : /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/fault.xml : Failed!
The file has been backed up into : NO_BACKUP_ON_WORKER.INFO
Deployment of synapse artifact failed. Error reading /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/fault.xml : Sequence Deployment from the file : /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/fault.xml : Failed.
org.apache.axis2.deployment.DeploymentException: Sequence Deployment from the file : /var/wso2am/repository/deployment/server/synapse-configs/default/sequences/fault.xml : Failed.
Task service starting in CLUSTERED mode...
Successfully Initialized Eventing on Registry

Bee Bee · Accepted Answer · 2016-08-23T08:10:38

There can be 2 reasons for this.

1) Using hostnames in <localMemberHost> and <members> sections in axis2.xml. Use IPs instead.

2) Not having the same value in store and publisher for <parameter name="domain">wso2.carbon.domain</parameter> in axis2.xml

Update: Attached configurations look fine. So another possibility is that this is a registry indexing problem. For that you can try reindexing.

Update <lastAccessTimeLocation> value of registry.xml in store node like this, and restart the server.

<lastAccessTimeLocation>/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime2</lastAccessTimeLocation>

No APIs published on the store in clustered mode - wso2 api manager 2.0.0

1 Answers