JCE: Exception while verifying X509 self-signed certificate

Question

We develop a custom JCE security provider for SSL/TLS.

One of our users is getting a server certificate verification failure on the client side. It's the usual "unable to find valid certification path to requested target" error. (Yes, the cert is in the truststore.)

Note: Although we are implementing a custom provider, we rely on the standard JCE providers for the trust manager, using javax.net.ssl.X509TrustManager.checkServerTrusted(X509Certificate[] chain, String) during the TLS handshake.

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:105)
        [snip]
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
        ...

As with all our users, they have the unlimited security policy jars installed.

The server certificate is self-signed (not CA). They use 127.0.0.1 as the hostname since they're just connecting to a backend process.

A default installation of the JRE/JDK works (using Sun/Oracle security providers). Using javax.net.debug output, I have confirmed that these successful connections use the same self-signed certificate.

However, when I hacked together some code to simply establish a connection, it works without issue using the same truststore, keystore, certificate, and JDK on the same machine. This uses the same verification function which makes the same call to X509TrustManager.checkServerTrusted() using the same certificate and the same authType string. I cannot for the life of me explain why checkServerTrusted() verifies the cert in this case but fails in the user's case.

Is it possible that there's some way to tweak the JCE such that the X509TrustManager would fail to verify this cert? Perhaps because it's self-signed, or because the common name is the name of the service and not a domain name? I see nothing in their JVM parameters or security properties to indicate this. But perhaps they're making some JCE call I don't know about which modifies X509TrustManager's behavior?

The javax.net.debug output for the cert is below. The common name is simply the name of our adopter's service. That seems fishy to me. However, it works with the default security providers.

***
Found trusted certificate:
[
[
  Version: V3
  Subject: CN=[snip], OU=[snip], O=[snip], L=Bangalore, ST=[snip], C=IN
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  modulus: [snip]
  public exponent: 65537
  Validity: [From: Wed Feb 17 14:45:40 IST 2016,
               To: Thu Nov 20 14:45:40 IST 2070]
  Issuer: CN=[snip], OU=[snip], O=[snip], L=Bangalore, ST=[snip], C=IN
  SerialNumber: [    5cf68160]

Certificate Extensions: 1
[1]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
[snip]
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
[snip]

]

After adding some additional logs, it's clear that the accepted issuers are being populated differently for whatever reason. In the failing case, a plethora of accepted issuers are returned from [trust-manager].getAcceptedIssuers[]; they do not include the problematic cert. In the passing case, only the problematic cert is included in the accepted issuers.

[edit 1] Corrected the certificate.

[edit 2] Corrected common name in the bold question.

[edit 3] Added accepted issuers paragraph

This problem is not name checking, because that occurs only after path validation is successful, and only for some applications e.g. HTTPS. However if/when it is done, your text says 'they' (the users?) use 127.0.0.1 but your trace shows the cert has CN=localhost; those are not the same name and won't match when tried, even though normally both are names for the same host and pseudo-interface. As to your actual problem, I can confirm selfsigned cert and hostname=localhost work fine, as you apparently did also. ... — dave_thompson_085
... I can't think of any JVM setting except (secprop) jdk.certpath.disabledAlgorithms, which should be the same for all apps in a given JRE, and shouldn't affect this case anyway. Best I can suggest is try running with sysprop java.security.debug=certpath and see if what it says helps at all. :-? — dave_thompson_085
I realize that 127.0.0.1 and localhost are different. However, in my test code, the connection is successfully established using that cert regardless of whether I used 127.0.0.1 or localhost, both with and without our provider. Also, I managed to put the wrong cert in the post. That's from a previous connection where we're acting as the server (which has no issues). I'll try certpath and see what that gives me. Thanks. — Andrew Michael Felsher
Here's something weird. java.security.debug=certpath doesn't seem to produce any output when using the default providers or when using my test code (i.e. all the scenarios that don't exhibit the exception). But I get a ton of output for the failing case (using the adopter's application with our security provider). — Andrew Michael Felsher
The debug feature prints directly on System.err so I can't see any reason for it to fail unless you do something odd like 2>/dev/null. But tm.getAcceptedIssusers() is the truststore contents so if that doesn't contain the desired cert(s) then your premise 'the cert is in the truststore' is wrong. Are you certain how the truststore being used is created? Does the relevant code create an explicit SSLContext and if so how, or use the default and if so check sysprops javax.net.ssl.trustStore* — dave_thompson_085

Andrew Michael Felsher Andrew Michael Felsher · Accepted Answer · 2017-08-02T14:35:13

It turns out that this is an issue with caching the trust manager.

With either the default providers or our custom provider, the trust manager is initially instantiated with the cacerts truststore.

With the default providers, a new trust manager is later instantiated using the javax.net.ssl.trustStore-specified truststore. Our custom provider just re-uses the previously instantiated cacerts trust manager.

Solution: Instantiate a new trust manager, which will respect javax.net.ssl.trustStore.

JCE: Exception while verifying X509 self-signed certificate

1 Answers