We have an OSGI application running on WebSphere Liberty profile which is expected to make HTTPS calls to different endpoints.
Some of these endpoints have shared their SSL certificates to be imported into the server truststore. Previously the certificates would be added to the truststore and the application restarted to pickup the changes.
This was the configuration being used for the truststore (the ${} are placeholders read from a properties file):
<keyStore id="defaultKeyStore" location="${keystore.location}"
password="${keystore.password}" type="${keystore.type}" />
<keyStore id="trustStore" location="${truststore.location}"
password="${truststore.password}"
type="${truststore.type}" />
<ssl clientAuthentication="false"
clientAuthenticationSupported="true"
id="defaultSSLConfig"
keyStoreRef="defaultKeyStore"
sslProtocol="SSL_TLSv2"
trustStoreRef="trustStore" />
<sslDefault sslRef="defaultSSLConfig" />
Recently this was modified to use the keystore polling for the truststore by making the following changes:
<keyStore id="trustStore" location="${truststore.location}"
password="${truststore.password}"
type="${truststore.type}" pollingRate="5s" updateTrigger="polled"/>
The attributes used are described here:
Keystore files can be reloaded by the server if the updateTrigger attribute is set to polled or mbean. If polled is enabled, then the server monitors the keystore file for changes based on the rate set in the pollingRate attribute.
Now, if I import a certificate into the truststore of the running server, I get the following message in the console:
[AUDIT ] CWPKI0811I: The keystore file resources\security\trust.jks has been modified. The keystore file will be reloaded so the updated keystore file can be used.
But the HTTPS call to the endpoint still fails with the certificate exception, until the server is restarted (after restart with no other changes the endpoint call succeeds, so the certificate itself is correct, and the endpoint is valid):
javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: unable to find valid certification path to requested target
at com.ibm.jsse2.k.a(k.java:15)
at com.ibm.jsse2.av.a(av.java:531)
at com.ibm.jsse2.D.a(D.java:68)
at com.ibm.jsse2.D.a(D.java:628)
at com.ibm.jsse2.E.a(E.java:803)
at com.ibm.jsse2.E.a(E.java:447)
at com.ibm.jsse2.D.r(D.java:139)
at com.ibm.jsse2.D.a(D.java:485)
at com.ibm.jsse2.av.a(av.java:717)
at com.ibm.jsse2.av.i(av.java:869)
at com.ibm.jsse2.av.a(av.java:19)
at com.ibm.jsse2.av.startHandshake(av.java:672)
at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:46)
at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:35)
Is the configuration I have used above incorrect or how should I configure the Liberty profile to correctly reload the certificates?
Some additional things I have tried:
- Get the default sslcontext using the
SSLContext.getDefault()and re-init it with the truststore - this works because Liberty itself replaces the original, immutable truststore with its own. However, if possible I would like to avoid this approach and use the Liberty standard one.
EDIT: Interesting, if I use the absolute path of the truststore in server.xml, things start to work. The relative path does not work.
With the relative path:
[11/5/18 13:17:07:870 IST] 00000084 id= com.ibm.ws.ssl.internal.KeystoreConfigurationFactory > performFileBasedAction Entry
[resources\security\trust.jks]
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 Clearing standard javax.net.ssl.SSLContext cache.
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.WSKeyStore 3 clearJavaKeyStore
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.WSKeyStore 3 clearJavaKeyStore
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.WSKeyStore 3 clearJavaKeyStore
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.WSKeyStore 3 clearJavaKeyStore
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.SSLConfigManager > resetDefaultSSLContext Entry
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.SSLConfigManager > getDefaultSSLConfig Entry
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.SSLConfigManager 3 getGlobalProperty -> com.ibm.ssl.defaultAlias=defaultSSLConfig
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.SSLConfigManager 3 defaultAlias: defaultSSLConfig
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.SSLConfigManager < defaultAlias not null, getDefaultSSLConfig for: defaultSSLConfig Exit
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.SSLConfigManager > keyStoreModified Entry
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.SSLConfigManager < keyStoreModified false Exit
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.SSLConfigManager 3 Modified keystore file are not part of the default SSL configuration.
[11/5/18 13:17:07:871 IST] 00000084 id= com.ibm.ws.ssl.config.SSLConfigManager < resetDefaultSSLContext Exit
[11/5/18 13:17:07:872 IST] 00000084 id= com.ibm.ws.ssl.internal.KeystoreConfigurationFactory A CWPKI0811I: The keystore file resources\security\trust.jks has been modified. The keystore file will be reloaded so the updated keystore file can be used.
[11/5/18 13:17:07:872 IST] 00000084 id= com.ibm.ws.ssl.internal.KeystoreConfigurationFactory < performFileBasedAction Exit
With the absolute path:
[11/5/18 13:11:32:720 IST] 00000086 id= com.ibm.ws.ssl.internal.KeystoreConfigurationFactory > performFileBasedAction Entry
[D:\programs\WebSphere\wlp-webProfile7-18.0.0.1\wlp\usr\servers\defaultServer\resources\security\trust.jks]
[11/5/18 13:11:32:723 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 Clearing standard javax.net.ssl.SSLContext cache.
[11/5/18 13:11:32:723 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 clearJavaKeyStore
[11/5/18 13:11:32:723 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 clearJavaKeyStore
[11/5/18 13:11:32:723 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 clearJavaKeyStore
[11/5/18 13:11:32:723 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 clearJavaKeyStore
[11/5/18 13:11:32:724 IST] 00000086 id= com.ibm.ws.ssl.config.SSLConfigManager > resetDefaultSSLContext Entry
[11/5/18 13:11:32:724 IST] 00000086 id= com.ibm.ws.ssl.config.SSLConfigManager > getDefaultSSLConfig Entry
[11/5/18 13:11:32:724 IST] 00000086 id= com.ibm.ws.ssl.config.SSLConfigManager 3 getGlobalProperty -> com.ibm.ssl.defaultAlias=defaultSSLConfig
[11/5/18 13:11:32:724 IST] 00000086 id= com.ibm.ws.ssl.config.SSLConfigManager 3 defaultAlias: defaultSSLConfig
[11/5/18 13:11:32:724 IST] 00000086 id= com.ibm.ws.ssl.config.SSLConfigManager < defaultAlias not null, getDefaultSSLConfig for: defaultSSLConfig Exit
[11/5/18 13:11:32:724 IST] 00000086 id= com.ibm.ws.ssl.config.SSLConfigManager > keyStoreModified Entry
[11/5/18 13:11:32:726 IST] 00000086 id= com.ibm.ws.ssl.config.SSLConfigManager < keyStoreModified true Exit
[11/5/18 13:11:32:726 IST] 00000086 id= com.ibm.ws.ssl.JSSEProviderFactory > getInstance: null Entry
[11/5/18 13:11:32:726 IST] 00000086 id= com.ibm.ws.ssl.JSSEProviderFactory < getInstance: com.ibm.ws.ssl.provider.IBMJSSEProvider@50d8b2eb Exit
[11/5/18 13:11:32:727 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider > setServerDefaultSSLContext Entry
[11/5/18 13:11:32:727 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider > getSSLContext Entry
null
[11/5/18 13:11:32:727 IST] 00000086 id= com.ibm.ws.ssl.config.ThreadContext 3 setOutboundConnectionInfoInternal :null
[11/5/18 13:11:32:727 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 outboundConnectionInfo: null
[11/5/18 13:11:32:727 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider > getSSLContextInstance Entry
[11/5/18 13:11:32:728 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider < getSSLContextInstance Exit
[11/5/18 13:11:32:728 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider > getKeyTrustManagers Entry
null
SSLConfig.toString() {
com.ibm.ssl.clientAuthenticationSupported=false
com.ibm.ssl.contextProvider=IBMJSSE2
config.displayId=keyStore[defaultKeyStore]
com.ibm.ssl.protocol=SSL_TLS
com.ibm.ssl.keyStoreReadOnly=false
com.ibm.ssl.alias=defaultSSLConfig
com.ibm.ssl.keyStoreCreateCMSStash=false
com.ibm.ssl.securityLevel=HIGH
com.ibm.ssl.trustStoreName=jvmTrustStore
com.ibm.ssl.trustStorePassword=********
service.pid=com.ibm.ws.ssl.keystore_21
com.ibm.ssl.trustManager=PKIX
com.ibm.ssl.validationEnabled=false
com.ibm.ssl.trustStoreInitializeAtStartup=false
com.ibm.ssl.keyManager=IbmX509
com.ibm.ssl.keyStoreFileBased=true
com.ibm.ssl.keyStoreType=jks
com.ibm.ssl.trustStoreFileBased=true
com.ibm.ssl.trustStoreCreateCMSStash=false
com.ibm.ssl.trustStore=D:/programs/WebSphere/wlp-webProfile7-18.0.0.1/wlp/usr/servers/defaultServer/resources/security/trust.jks
config.overrides=true
com.ibm.ssl.daysBeforeExpireWarning=60
sslRef=defaultSSLConfig
id=defaultKeyStore
config.id=com.ibm.ws.ssl.keystore[defaultKeyStore]
com.ibm.ssl.clientAuthentication=false
com.ibm.ssl.keyStore=resources/security/key.jks
com.ibm.ssl.trustStoreReadOnly=false
config.source=file
alias=defaultSSLConfig
com.ibm.ssl.tokenEnabled=false
com.ibm.ssl.keyStoreName=defaultKeyStore
com.ibm.ssl.keyStorePassword=********
com.ibm.ssl.keyStoreInitializeAtStartup=false
service.factoryPid=com.ibm.ws.ssl.keystore
com.ibm.ssl.trustStoreType=jks
}
[11/5/18 13:11:32:728 IST] 00000086 id= com.ibm.ws.ssl.config.KeyStoreManager 3 Returning a keyStore for name: jvmTrustStore
[11/5/18 13:11:32:728 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore > do_getKeyStore Entry
false
false
[11/5/18 13:11:32:728 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 Initializing KeyStore: jvmTrustStore
[11/5/18 13:11:32:729 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 Password was not decoded.
[11/5/18 13:11:32:729 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 File path for store: D:/programs/WebSphere/wlp-webProfile7-18.0.0.1/wlp/usr/servers/defaultServer/resources/security/trust.jks
[11/5/18 13:11:32:729 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 Loading keyStore (filebased)
[11/5/18 13:11:32:729 IST] 00000086 id= com.ibm.ws.ssl.JSSEProviderFactory > getInstance: null Entry
[11/5/18 13:11:32:729 IST] 00000086 id= com.ibm.ws.ssl.JSSEProviderFactory < getInstance: com.ibm.ws.ssl.provider.IBMJSSEProvider@50d8b2eb Exit
[11/5/18 13:11:32:729 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 KeyStore.getInstance(jks, null)
[11/5/18 13:11:32:731 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 alias: p13
[11/5/18 13:11:32:731 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 alias: p12
[11/5/18 13:11:32:731 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 do_getKeyStore (initialized)
[11/5/18 13:11:32:731 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore < do_getKeyStore Exit
java.security.KeyStore@7be5d76a
[11/5/18 13:11:32:731 IST] 00000086 id= com.ibm.ws.ssl.config.KeyStoreManager 3 Returning a keyStore for name: defaultKeyStore
[11/5/18 13:11:32:732 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore > do_getKeyStore Entry
false
false
[11/5/18 13:11:32:732 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 Initializing KeyStore: defaultKeyStore
[11/5/18 13:11:32:732 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 Password was not decoded.
[11/5/18 13:11:32:732 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 File path for store: resources/security/key.jks
[11/5/18 13:11:32:732 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 Loading keyStore (filebased)
[11/5/18 13:11:32:732 IST] 00000086 id= com.ibm.ws.ssl.JSSEProviderFactory > getInstance: null Entry
[11/5/18 13:11:32:732 IST] 00000086 id= com.ibm.ws.ssl.JSSEProviderFactory < getInstance: com.ibm.ws.ssl.provider.IBMJSSEProvider@50d8b2eb Exit
[11/5/18 13:11:32:733 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 KeyStore.getInstance(jks, null)
[11/5/18 13:11:32:735 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 alias: default
[11/5/18 13:11:32:735 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 do_getKeyStore (initialized)
[11/5/18 13:11:32:735 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore < do_getKeyStore Exit
java.security.KeyStore@941dcba8
[11/5/18 13:11:32:735 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 getLocation -> D:/programs/WebSphere/wlp-webProfile7-18.0.0.1/wlp/usr/servers/defaultServer/resources/security/trust.jks
[11/5/18 13:11:32:735 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 Using trust store: D:/programs/WebSphere/wlp-webProfile7-18.0.0.1/wlp/usr/servers/defaultServer/resources/security/trust.jks
[11/5/18 13:11:32:736 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 getTrustManagerFactory.getInstance(PKIX, IBMJSSE2)javax.net.ssl.TrustManagerFactory@c99b19d6
[11/5/18 13:11:32:736 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 certStoreHost: null
[11/5/18 13:11:32:736 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 certStorePort: 389
[11/5/18 13:11:32:736 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 trustManagerAlgorithm: PKIX
[11/5/18 13:11:32:736 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 getLocation -> resources/security/key.jks
[11/5/18 13:11:32:736 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 Using software keystore: resources/security/key.jks
[11/5/18 13:11:32:736 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 getKeyManagerFactory.getInstance(IbmX509, IBMJSSE2) javax.net.ssl.KeyManagerFactory@df035ba8
[11/5/18 13:11:32:737 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 Password was not decoded.
[11/5/18 13:11:32:737 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 Entering synchronized block around key manager factory init.
[11/5/18 13:11:32:739 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 Exiting synchronized block around key manager factory init.
[11/5/18 13:11:32:740 IST] 00000086 id= com.ibm.ws.ssl.core.WSX509KeyManager > WSX509KeyManager Entry
[11/5/18 13:11:32:740 IST] 00000086 id= com.ibm.ws.ssl.core.CertMappingKeyManager > <init> Entry
[11/5/18 13:11:32:740 IST] 00000086 id= com.ibm.ws.ssl.core.CertMappingKeyManager > parseSSLCertFile Entry
[11/5/18 13:11:32:740 IST] 00000086 id= com.ibm.ws.ssl.core.CertMappingKeyManager < parseSSLCertFile Exit
[11/5/18 13:11:32:740 IST] 00000086 id= com.ibm.ws.ssl.core.CertMappingKeyManager < <init> Exit
[11/5/18 13:11:32:740 IST] 00000086 id= com.ibm.ws.ssl.config.KeyStoreManager 3 Returning a keyStore for name: defaultKeyStore
[11/5/18 13:11:32:740 IST] 00000086 id= com.ibm.ws.ssl.core.WSX509KeyManager < WSX509KeyManager Exit
[11/5/18 13:11:32:741 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 Initializing WSX509KeyManager.
null
null
null
[11/5/18 13:11:32:741 IST] 00000086 id= com.ibm.ws.ssl.config.WSKeyStore 3 getLocation -> D:/programs/WebSphere/wlp-webProfile7-18.0.0.1/wlp/usr/servers/defaultServer/resources/security/trust.jks
[11/5/18 13:11:32:741 IST] 00000086 id= com.ibm.ws.ssl.core.WSX509TrustManager > WSX509TrustManager Entry
null
D:/programs/WebSphere/wlp-webProfile7-18.0.0.1/wlp/usr/servers/defaultServer/resources/security/trust.jks
[11/5/18 13:11:32:742 IST] 00000086 id= com.ibm.ws.ssl.core.WSX509TrustManager < WSX509TrustManager Exit
[11/5/18 13:11:32:742 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider < getKeyTrustManagers Exit
[11/5/18 13:11:32:743 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 SSLContext cache size: 1
[11/5/18 13:11:32:743 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider < getSSLContext -> (new) Exit
[11/5/18 13:11:32:743 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider 3 Default SSLContext set to defaultSSLConfig
[11/5/18 13:11:32:743 IST] 00000086 id= com.ibm.ws.ssl.provider.AbstractJSSEProvider < setServerDefaultSSLContext Exit
[11/5/18 13:11:32:743 IST] 00000086 id= com.ibm.ws.ssl.config.SSLConfigManager < resetDefaultSSLContext Exit
[11/5/18 13:11:32:743 IST] 00000086 id= com.ibm.ws.ssl.internal.KeystoreConfigurationFactory A CWPKI0811I: The keystore file D:\programs\WebSphere\wlp-webProfile7-18.0.0.1\wlp\usr\servers\defaultServer\resources\security\trust.jks has been modified. The keystore file will be reloaded so the updated keystore file can be used.
[11/5/18 13:11:32:744 IST] 00000086 id= com.ibm.ws.ssl.internal.KeystoreConfigurationFactory < performFileBasedAction Exit
