Using self-signed SSL certificate works, but CA signed certificate results in handshake alert failure 40 in response to client hello

1
votes

So far, I've used openssl, sslyze, keystore utilities, and some of the standard Windows diagnostics commands to try and characterize this problem. The summary is that as soon as I attempt to use a keystore that has the CA signed certificate in it, I get an immediate handshake failure 40 after the client hello. The connection never gets as far as a server hello with the CA cert in place.

On the same machine (Windows Server 2012), if I use a self signed certificate, the connection works as expected.

Openssl shows the handshake failure with the CA certificate in place.

sslyze shows all ciphers rejected with either no ciphers available or TLS / Alert handshake failure.

The fact that this is successful with a self-signed certificate tends to indicate that the "basics" are there, that the client and server have the necessary ciphers and so on to connect, and that server.xml is configured properly.

There were no error messages when importing the certificates, and everything seems to be configured correctly with the CA certs based on keytool checks. There are no error messages given during server start to indicate any problems processing the certificate.

Is there something I should be looking more closely at in regards to the keystore with the CA cert that could cause this complete rejection of the client hello? How would a different keystore or CA certs within aaffect the earliest steps of the handshake?

Thank you for information provided.

1
ssl
Assuming 'keytool' means Java, this behavior is typical of a "keystore" file that does not actually contain a key. When you say the keystore "has the CA-signed cert", does it have the CA-signed cert in the same entry, of type privateKey, used to generate the cert request (CSR)? Further, all public CAs today issue certs that need at least one "chain" or "intermediate" cert (sometimes more) and you must have that as well (although it/they can start out in a different entry/ies of type trustedCert). – dave_thompson_085

1 Answers

0
votes

I assume that you are talking about a server side certificate and a server side key store, since the error happens at a state where no client certificates are yet involved. If the server is able to send the self-signed certificate to the client, but is not able to send the CA signed certificate to the client, then it must be something wrong with the certificate you are trying to send or that the certificate can not be used together with the ciphers offered by the client.

Since the problem is obviously on the server side you should first check all logs written on the server side for hints what the error might be. Typical problems are non-existing files, wrong files, password protected client key w/o providing a password or that the key does not belong to the certificate.