IBM WebSphere Liberty working with IHS Server Problems with Certificate(s)

Question

I have a liberty server and an IHS server (both on different hosts). This is on rhel7.

I've followed a number of guides that I found that walk thru the setup of keystores and sharing of certs but still run into issues. For reference, I tried the methods detailed https://jazz.net/wiki/bin/view/Deployment/CreateIHSPLUGINFORLIBERTYPROFILE and https://www.ibm.com/support/knowledgecenter/en/SSEQTJ_9.0.5/com.ibm.websphere.ihs.doc/ihs/tihs_install_config_liberty.html.

A quick summary of what I have done this past attempt... On the liberty server I added the following to my server.xml to add a plugin include (to keep things easier to read):

<include location="${server.config.dir}/plugin-join-include.xml" />

And the plugin-join-include.xml:

        <featureManager>
                <feature>ssl-1.0</feature>
        </featureManager>

        <keyStore id="libertyKey" password="xxxx"
              location="${server.config.dir}/resources/security/libkey.jks"/>

        <pluginConfiguration webserverPort="10500" webserverSecurePort="10447"
                pluginInstallRoot="/opt/IBM/wasadmin/Plugins"
                sslKeyRingLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"
                sslStashFileLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"
                sslCertlabel="libertyKey"/>

On my IHS server the path for the root Plugin directory is as indicated above in the file at /opt/IBM/wasadmin/Plugins and the plugin-key.kdb and plugin-key.sth are located in the locations listed above.

I reboot the liberty server and it generated the plugin-cfg.xml in the logs/state directory:

<?xml version="1.0" encoding="UTF-8"?><!--HTTP server plugin config file for app generated on 2020.03.04 at 12:32:02 UTC-->
<Config ASDisableNagle="false" AcceptAllContent="false" AppServerPortPreference="HostHeader" ChunkedResponse="false" ConfigHash="1077723051" FIPSEnable="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="false" TrustedProxyEnable="false" VHostMatchingCompat="false">
   <Log LogLevel="Error" Name="/opt/IBM/wasadmin/Plugins/logs/webserver1/http_plugin.log"/>
   <Property Name="ESIEnable" Value="true"/>
   <Property Name="ESIMaxCacheSize" Value="1024"/>
   <Property Name="ESIInvalidationMonitor" Value="false"/>
   <Property Name="ESIEnableToPassCookies" Value="false"/>
   <Property Name="PluginInstallRoot" Value="/opt/IBM/wasadmin/Plugins"/>
<!-- Configuration generated using httpEndpointRef=defaultHttpEndpoint-->
<!-- The default_host contained only aliases for endpoint defaultHttpEndpoint.
         The generated VirtualHostGroup will contain only configured web server ports:
                webserverPort=10500
                webserverSecurePort=10447 -->
   <VirtualHostGroup Name="default_host">
      <VirtualHost Name="*:10500"/>
      <VirtualHost Name="*:10447"/>
   </VirtualHostGroup>
   <ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="affms_default_node_Cluster" PostBufferSize="0" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1">
      <Server CloneID="412b3187-16c4-41b0-86e8-1e327c1c6b1b" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="default_node_affms" ServerIOTimeout="900" WaitForContinue="false">
         <Transport Hostname="libertyhost" Port="10500" Protocol="http"/>
         <Transport Hostname="libertyhost" Port="10447" Protocol="https">
            <Property Name="keyring" Value="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"/>
            <Property Name="stashfile" Value="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"/>
            <Property Name="certLabel" Value="libertyKey"/>
         </Transport>
      </Server>
      <PrimaryServers>
         <Server Name="default_node_app"/>
      </PrimaryServers>
   </ServerCluster>
   <UriGroup Name="default_host_app_default_node_Cluster_URIs">
      <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/app/ui/*"/>
      <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/adminCenter/*"/>
   </UriGroup>
   <Route ServerCluster="app_default_node_Cluster" UriGroup="default_host_app_default_node_Cluster_URIs" VirtualHostGroup="default_host"/>

I copy the plugin-cfg.xml over to the IHS server at /opt/IBM/wasadmin/Plugins/config/webserver1/plugin-cfg.xml

The plugin stuff works only the SSL portion between IHS and the app server give me issues.

On the IHS server I have created the plugin-key.kdb keystore with stash plugin-key.sth and imported the cert I extracted from the liberty server /opt/IBM/wlp/usr/servers/app/resources/security/key.p12 the default personal cert and copy it to the IHS server and add the default liberty cert to the plugin-key.kdb keystore.

At the bottom of the httpd.conf is:

LoadModule was_ap22_module /opt/IBM/wasadmin/Plugins/bin/64bits/mod_was_ap22_http.so
WebSpherePluginConfig /opt/IBM/wasadmin/Plugins/config/webserver1/plugin-cfg.xml

I bounce the apachectl and go to the url for the application on port 8443 https://hostname:8443/app/ui and get a "500 Internal Server Error".

The http_plugin.log plugin logs show:

[04/Mar/2020:18:23:48.31652] 00002f26 3cff9700 - ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414) PARTNER CERTIFICATE DN=CN=hostname,OU=app,O=ibm,C=us, Serial=xxxxxxxxxxx
[04/Mar/2020:18:23:48.31655] 00002f26 3cff9700 - ERROR: Ensure correct certificate is marked as default certificate in plugin-key.kdb.                           Consult documentation regarding Administering application security and Securing communications for more information.
[04/Mar/2020:18:23:48.31657] 00002f26 3cff9700 - ERROR: Last validation error [575010]: GSKVAL_ERROR_NO_CHAIN_BUILT
[04/Mar/2020:18:23:48.31658] 00002f26 3cff9700 - ERROR: Subject [[Class=]GSKVALMethod::PKIX[Issuer=]OU=memberRoot,O=xxxx-xxx-xxx-xxx-xxxxxxxx,DC=com.ibm.ws.collective[#=]082d6a83e5ec[Subject=]CN=hostname,OU=app,O=ibm,C=us] failed certificate validation
[04/Mar/2020:18:23:48.31659] 00002f26 3cff9700 - ERROR: X509 Certificate validation log: [[Class=]GSKVALMethod::PKIX[Time=]2020:3:4:18:23:48.316[buildChain=][Error=]GSKVAL_ERR_NO_CHAIN_BUILT[Info=]OU=memberRoot,O=xxx-xxx-xxx-xxx-xxx,DC=com.ibm.ws.collective[Cert=][Issuer=]OU=memberRoot,O=xxx-xxx-xxx-xxx-xxxx,DC=com.ibm.ws.collective[#=]082d6a83e5ec[Subject=]CN=hostname,OU=app,O=ibm,C=us[=Cert][=buildChain]^M
]
[04/Mar/2020:18:23:48.31664] 00002f26 3cff9700 - ERROR: ws_common: websphereGetStream: Could not open stream
[04/Mar/2020:18:23:48.31667] 00002f26 3cff9700 - ERROR: ws_common: websphereExecute: Failed to create the stream
[04/Mar/2020:18:23:48.31668] 00002f26 3cff9700 - ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'default_node_app' on host 'hostname:10447'; will try another one
[04/Mar/2020:18:23:48.31669] 00002f26 3cff9700 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
[04/Mar/2020:18:23:48.31670] 00002f26 3cff9700 - ERROR: ESI: getResponse: failed to get response: rc = 2
[04/Mar/2020:18:23:48.31671] 00002f26 3cff9700 - ERROR: [xxx.xx.xx.17://app/ui/] ws_common: websphereHandleRequest: Failed to handle request rc=2

Thank you!

Answer 1

You said "imported" I assume this means an "add" operation in ikeyman/gskcmd/gskcapicmd? The two verbs in these tools have a different meaning.

I suspect you added the issued certificate (CN=hostname...) instead of the CA that issued the certificate (memberroot). The debug information in the WAS Plugin message implies that it couldn't chase the certificate up to a trusted root (as opposed to finding some X509 error w/ the root it had access to).

But, it could also be that you imported the "wrong" certificate with cn=memberroot. WebSphere does not always add the necessary Subject Key Identifier/Authority Key Identifier to be sure, so check carefully.

Answer 2

Ok, I got this working finally. It seems having a collective setup which used DefaultKeyStore id was somehow causing conflict with my Liberty plugin keystore.

I ended up re-creating everything (plugin-key.kdb keystore and the liberty plugin keystore which I named LibertyKeystore.jks). I dumped the collective stuff (not needed at this time anyways) and made an include as above called plugin-join-include.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<server description="IHS plugin join include file">

        <featureManager>
                <feature>ssl-1.0</feature>
        </featureManager>

        <keyStore id="defaultKeyStore" password="password"
                location="${server.config.dir}/resources/security/LibertyKeystore.jks" />

        <pluginConfiguration webserverPort="80" webserverSecurePort="8443"
                pluginInstallRoot="/opt/IBM/wasadmin/Plugins"
                sslKeyRingLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"
                sslStashFileLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"
                sslCertlabel="LibertyKeystore"/>

</server>

I exchanged keys between the liberty server LibertyKeystore.jks and the IHS plugin-key.kdb keystores and pushed the newly generated plugin to the IHS server and bounced everything and it all seems to function without issue.

I do now notice in the /opt/IBM/wasadmin/HTTPServer/logs the error log streams:

[Wed Mar 04 21:32:28 2020] [error] [client xxx.xx.xx.17:65261] [7ff5a0000910] [19831] SSL0279E: SSL Handshake Failed due to fatal alert from client. Client sent fatal alert [level 2 (fatal), description 46 (certificate_unknown)]  [xxx.xx.xx.17:65261 -> xxx.xx.xx.87:8443] [21:32:28.000934048] 0ms

The access logs show 200's. Maybe a non plugin "issue" at this point.