Oracle BPEL (Java 8U92) invoking REST service using HTTPS gives SSL Handshake Exception

Question

I have setup a basic weblogic domain with oracle SOA 12c for developing a composite which can invoke Apples APN services. Apple APN requires an HTTP2 connection encrypted with TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 and TLS 1.2.

A JKS truststore has been configured and loaded with the certificates of the root, intermediate and server certificate.

geotrustrootca, Jun 21, 2016, trustedCertEntry, 
Certificate fingerprint (SHA1): DE:28:F4:A4:FF:E5:B9:2F:A3:C5:03:D1:A3:49:A7:F9:96:2A:82:12
serverc_ss_cert, Jun 21, 2016, trustedCertEntry, 
Certificate fingerprint (SHA1): 73:C4:A9:4E:E8:1B:14:58:7B:64:47:02:73:01:15:3E:88:E8:E8:66
appledevpush, Jun 21, 2016, trustedCertEntry, 
Certificate fingerprint (SHA1): CC:18:A5:75:04:74:3A:3B:72:D7:A5:07:F2:CD:E4:83:51:11:34:CB
appleintermediate, Jun 21, 2016, trustedCertEntry, 
Certificate fingerprint (SHA1): 8E:83:21:CA:08:B0:8E:37:26:FE:1D:82:99:68:84:EE:B5:F0:D6:55

The setDomainEnv.sh has been altered by adding the java property -Djavax.net.ssl.trustStore=/u01/data/keystores/truststore.jks

When testing a simple BPEL composite making a REST call to Apple's APN service if fails with a SSLHandshakeException:

[ACTIVE] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)', SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
[ACTIVE] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)', WRITE: TLSv1.2 Alert, length = 2
[ACTIVE] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)', called closeSocket()
[ACTIVE] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)', handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

When debugging the SSL negotiation is see;

%% No cached client session
*** ClientHello, TLSv1.2

....

*** ServerHello, TLSv1.2
RandomCookie:  GMT: 1922117017 bytes = { 236, 133, 59, 43, 182, 3, 165, 71, 241, 54, 240, 145, 222, 41, 200, 242, 63, 237, 253, 77, 188, 235, 187, 177, 245, 173, 53, 232 }
Session ID:  {119, 250, 96, 4, 116, 33, 211, 17, 47, 213, 227, 158, 164, 107, 14, 73, 157, 194, 0, 104, 54, 237, 0, 58, 229, 225, 158, 2, 29, 159, 79, 171}
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

...

%% Initialized:  [Session-7, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
** TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
[ACTIVE] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)', READ: TLSv1.2 Handshake, length = 2576
*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: C=US, ST=California, O=Apple Inc., OU=management:idms.group.533599, CN=api.development.push.apple.com
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

...

chain [1] = [
[
  Version: V3
  Subject: C=US, O=Apple Inc., OU=Certification Authority, CN=Apple IST CA 2 - G1
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

...

***
%% Invalidated:  [Session-7, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
[ACTIVE] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)', SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
[ACTIVE] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)', WRITE: TLSv1.2 Alert, length = 2
[ACTIVE] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)', called closeSocket()
[ACTIVE] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)', handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

My conclusion is that the server certificate for server N=api.development.push.apple.com is accepted but the certificate of the intermediate CA CN=Apple IST CA 2 - G1 is rejected.

The issuer of CN=Apple IST CA 2 - G1 is CN=GeoTrust Global CA, O=GeoTrust Inc., C=US with serial number 023a74. This certificate is also loaded in the truststore;

Alias name: geotrustrootca Creation date: Jun 21, 2016 Entry type: trustedCertEntry

Owner: CN=GeoTrust Global CA, O=GeoTrust Inc., C=US
Issuer: CN=GeoTrust Global CA, O=GeoTrust Inc., C=US
Serial number: 23456
Valid from: Tue May 21 06:00:00 CEST 2002 until: Sat May 21 06:00:00 CEST 2022
Certificate fingerprints:
     MD5:  F7:75:AB:29:FB:51:4E:B7:77:5E:FF:05:3C:99:8E:F5
     SHA1: DE:28:F4:A4:FF:E5:B9:2F:A3:C5:03:D1:A3:49:A7:F9:96:2A:82:12
     SHA256: FF:85:6A:2D:25:1D:CD:88:D3:66:56:F4:50:12:67:98:CF:AB:AA:DE:40:79:9C:72:2D:E4:D2:B5:DB:36:A7:3A
     Signature algorithm name: SHA1withRSA
     Version: 3

Any ideas (if my conclusion is correct) why the intermediate certificate is rejected or how to debug this further ? When opening the URI to APN using an browser and inspect the certificates, i get the same as in the truststore.

== Update 1 ==

Tried to connect using curl. First exported the certificates from the truststore to /u01/data/keystores

$keytool -keystore truststore.jks -exportcert -alias geotrustrootca | openssl x509 -inform der -text > geotrustrootca.pem
 $keytool -keystore truststore.jks -exportcert -alias appledevpush | openssl x509 -inform der -text > appledevpush.pem
$keytool -keystore truststore.jks -exportcert -alias appleintermediate | openssl x509 -inform der -text > appleintermediate.pem

Then tried to connect using curl

$ curl --capath /u01/data/keystores --verbose  https://api.development.push.apple.com/3/device/
* About to connect() to api.development.push.apple.com port 443 (#0)
*   Trying 17.172.238.203... connected
* Connected to api.development.push.apple.com (17.172.238.203) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* failed to load '/u01/data/keystores/identitykeystore.jks' from CURLOPT_CAPATH
* failed to load '/u01/data/keystores/appledevpush.cer' from CURLOPT_CAPATH
* failed to load '/u01/data/keystores/geotrustrootca.cer' from CURLOPT_CAPATH
* failed to load '/u01/data/keystores/truststore.jks' from CURLOPT_CAPATH
* failed to load '/u01/data/keystores/yum-oracle-8v1ncO' from CURLOPT_CAPATH
* failed to load '/u01/data/keystores/vm0010.localdomain-rootCA.der' from CURLOPT_CAPATH
* failed to load '/u01/data/keystores/appleintermediate.cer' from CURLOPT_CAPATH
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: /u01/data/keystores
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*   subject: C=US,ST=California,O=Apple Inc.,OU=management:idms.group.533599,CN=api.development.push.apple.com
*   start date: Jun 19 01:49:43 2015 GMT
*   expire date: Jul 18 01:49:43 2017 GMT
*   common name: api.development.push.apple.com
*   issuer: C=US,O=Apple Inc.,OU=Certification Authority,CN=Apple IST CA 2 - G1
> GET /3/device/ HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: api.development.push.apple.com
> Accept: */*
> 
* Connection #0 to host api.development.push.apple.com left intact
* Closing connection #0
@@�HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 474554202f332f6465766963652f20485454502f312e310d

So the certificates are correct.

== update 2 ==

Re-created the truststore once more. Acquired the pem files by opening the url https://api.development.push.apple.com/3/device/ in netscape and saving the certificates in pem format.

Imported the certificates in a new truststore.jks

for file in ls -1 *.der ; do keytool -importcert -keystore truststore.jks -file $file -storepass welcome1 -noprompt -alias $file ; done

No Joy...

== Update 3 ==

When restarting the Managed Server is now obvserved in the server.out that the default self signed certificates are loaded also.

<Jul 11, 2016 10:15:50 PM CEST> <Warning> <oracle.soa.healthcheck> <BEA-000000> <On startup, health check id 881 failed for category 'Startup'. Ran 6 checks. Number of failures=1, errors=1, warnings=0.>
adding as trusted cert:
  Subject: CN=CertGenCA, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  Issuer:  CN=CertGenCA, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  Algorithm: RSA; Serial number: 0x40044886c441ef3b643a8066409afca0
  Valid from Sat Dec 01 04:07:51 CET 2012 until Thu Dec 02 04:07:51 CET 2032

adding as trusted cert:
  Subject: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  Issuer:  CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  Algorithm: RSA; Serial number: 0x234b5559d1fa0f3ff5c82bdfed032a87
  Valid from Thu Oct 24 17:54:45 CEST 2002 until Tue Oct 25 17:54:45 CEST 2022

javax.net.ssl.trustStore is set to the custom truststore. Also in weblogic the Identity and Truststore locations have been set to the custom JKS stores.

Even after removing the DemoTrust.jks from the $WLS_HOME/lib the certs are added as trusted certs. I am currently clueless where these do come from.

== Update 4 ==

$DOMAIN_HOME/security contained the DemoIdentity.jks. After removing this file and restarting the Managed Server the Demo trusted certs are not loaded anymore.

== Update 5 ==

Verified the truststore against the targeted url SAS SSL/TLS diagnosics tool. To make sure the truststore is setup correctly.

As pointed out by user2351802 the OPSS Keystore has to be used instead of the JKS keystore defined in the Java property javax.net.ssl.trustStore.

In SOA Suite 10g/11g the standard way secure transport when invoking external webservices from composites using one-way SSL was by creating a JKS truststore and specify the location of that truststore in the javax.net.ssl.trustStore Java property. It is even still documented in the SOA Suite 12.2.1.1 Admin Guide to use the JKS truststore for maken one-way SSL connections from SOA composites. The Oracle WebLogic Server 12.2.1.1.0 Admin Guide mentions the (new) OPSS Keystore and refers to the document "Securing Applications with Oracle Platform Security Services" for using and configuring the new KSS OPSS Keystore. Although it mentions the procedure for setting up one-way SSL explicitly for LDAP it seems the new common way for FMW applications.

After cleaning up the identity and truststore in weblogic and only adding the server certificate to the identity store (thus no trusted certs anymore in the JKS truststore) and adding the root certificate to the OPSS system/trust it worked!

What does the jdk.certpath.disabledAlgorithms line in the file $JAVA_HOME/lib/security/java.security contain? — Andrew Henle
It contains; jdk.certpath.disabledAlgorithms=MD2, MD5, RSA keySize < 1024 — pcvnes
Can you get anything else to successfully establish an SSL connection, such as OpenSSL with openssl s_client -connect ...? — Andrew Henle
Can connect with my browser and curl, will try with openssl later today — pcvnes
openssl s_client -host api.development.push.apple.com -port 443 CONNECTED(00000003) depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA verify return:1 depth=1 CN = Apple IST CA 2 - G1, OU = Certification Authority, O = Apple Inc., C = US verify return:1 depth=0 CN = api.development.push.apple.com, OU = management:idms.group.533599, O = Apple Inc., ST = California, C = US ... Verify return code: 0 (ok) --- — pcvnes

user2351802 user2351802 · Accepted Answer · 2016-07-19T15:27:04

OK. It will sound wierd but it will work. We have the same SOA 12c setup but we are using Standard Java Trust keystore for managed SOA server.

I can see that you have modified setDomainEnv.sh to specify /u01/data/keystores/truststore.jks as your keystore.

In theory if the root cert is there in the cacerts in my case and truststore.jks in your case it should work. I can confirm that the SOAP services work fine with the keystore.

Somehow invoking REST service through REST adapter was failing due to cert error, same as yours.

Here's what made it to work:
Login to EM
Weblogic Domain -> Security -> Keystore
Select System (stripe) -> trust -> Hit the manage button
Here import the root cert of geotrustrootca.

Bounce the SOA server. Test your service. It should work fine.

What I dont understand is: System (stripe) -> trust = This is preconfigured as a demo trust store when you configure a domain. I have already change the keystore setting for managed server to use cacerts. Somehow it looks like this kss trust store is still referenced somewhere. The question is where?

Do share just in case you figure this one out. In the meantime the solution will get you going.

Oracle BPEL (Java 8U92) invoking REST service using HTTPS gives SSL Handshake Exception

1 Answers