WSO2 Authentication failure on gateway node running WSO2 AM 2.1.0

Question

After working with WSO2 AM 1.10.0, we're now evaluating AM 2.1.0 for our WSO2 cluster. The cluster has two types of nodes:

• A manager node: a complete set of WSO2 AM products.
• A worker node: which runs only the gateway (-DworkerNode=true -Dprofile=gateway-worker)

At the manager node, everything works ok. I am able to create a tenant (0000s7.com) and an API, and run it (I defined the GET verb to be unauthenticated, so no subscription is needed):

    $ curl http://localhost:8280/t/0000s7.com/ofer1/1.0.0
    { ...valid response... }

At the worker node:

When I’m working with WSClient, I get the following error in the log:

TID: [-1234] [] [2018-02-06 07:26:48,893] ERROR {org.apache.axis2.transport.http.AxisServlet} -   
{org.apache.axis2.transport.http.AxisServlet} java.lang.NullPointerException
        at org.apache.synapse.rest.RESTUtils.getFullRequestPath(RESTUtils.java:80)
        at org.apache.synapse.rest.API.canProcess(API.java:223)
        at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:97)
        at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:69)
        at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:304)

I also tried ThriftClient, with it I get a different error in the log:

   TID: [1] [] [2018-02-06 08:01:29,527] @0000s7.com [1] [AM] 
WARN {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClien    t} -  
Login failed.. Authenticating again..{org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient}

In both cases, API call ends with

<ams:fault xmlns:ams="http://wso2.org/apimanager/security">
<ams:code>900900</ams:code><ams:message>Unclassified Authentication Failure</ams:message>
<ams:description>Error while accessing backend services for API key validation</ams:description>
</ams:fault>

Quick summary of changes. Currently I left the WSClient version to run:

Manager Node (internal IP 10.1.0.7)

• ./repository/conf/api-manager.xml

          <KeyValidatorClientType>WSClient</KeyValidatorClientType>
          <ThriftClientConnectionTimeOut>10000</ThriftClientConnectionTimeOut>
          <ThriftClientPort>10397</ThriftClientPort>
  
          <EnableThriftServer>false</EnableThriftServer>
          <ThriftServerHost>10.1.0.7</ThriftServerHost>
          <ThriftServerPort>10397</ThriftServerPort>
  

• ./repository/conf/carbon.xml

      <HostName>10.1.0.7</HostName>
      <MgtHostName>10.1.0.7</MgtHostName>

• ./repository/conf/registry.xml
• ./repository/conf/user-mgt.xml

Minor changes to adopt to the MySQL DB

• ./repository/conf/axis2/axis2.xml

            <property name="subDomain" value="mgt"/>
            <property name="port.mapping.80" value="9763"/>
            <property name="port.mapping.443" value="9443"/>

• ./repository/conf/datasources/master-datasources.xml

Changed to work with RDS using MySQL. 
I'm using four databases:
o regdb_21 (the “shared” registry”
o apimgtdb_21
o userdb_21
o msgbrokerdb_21
In addition, stats DB and local registry are maintained in H2.

Worker Node (Internal IP 10.1.0.150)

• ./bin/wso2server.sh

     -DworkerNode=true \
     -Dprofile=gateway-worker \

• ./repository/conf/api-manager.xml

        <KeyValidatorClientType>WSClient</KeyValidatorClientType>
        <ThriftClientPort>10397</ThriftClientPort>
        <ThriftClientConnectionTimeOut>10000</ThriftClientConnectionTimeOut>

        <ThriftServerHost>10.1.0.7</ThriftServerHost>
        <ThriftServerPort>10397</ThriftServerPort>
        <EnableThriftServer>false</EnableThriftServer>
        <JMSConnectionDetails>
            <Enabled>false</Enabled>
            …

   Plus some changes of JMS IP to the IP address of the manager (10.1.0.7)

• ./repository/conf/carbon.xml

      <HostName>10.1.0.150</HostName>
      <MgtHostName>10.1.0.150</MgtHostName>

• ./repository/conf/registry.xml
• ./repository/conf/user-mgt.xml

            Minor changes to adopt to the MySQL DB

• ./repository/conf/axis2/axis2.xml

         <property name="port.mapping.80" value="9763"/>
         <property name="port.mapping.443" value="9443"/>

• ./repository/conf/datasources/master-datasources.xml

        Same as in manager

• ./repository/conf/broker.xml
• ./repository/conf/event-broker.xml
• ./repository/conf/jndi.properties
• ./repository/conf/messaging-event-broker.xml

  Changes in IP from localhost to 10.1.0.7

Complete logs

• Manager: http://thatsitradio.com/logs1/manager.html
• Worker: http://thatsitradio.com/logs1/worker.html

Answer 1

Since the Keymanager components are in manager node, you have to change <APIKeyValidator> of the worker to point to manager node.

If you have -Dorg.wso2.ignoreHostnameVerification=true \ in wso2server.sh file, that should ignore the hostname validation.