Restrict TLS mutual authentication to specific TLS certificates

Question

There is a PKI with a single CA issuing all the x509 certificates in my network. So on my network all the servers and clients possess a certificate from my CA stored in their corresponding keystore together with the private key. Each server and client has also the CA certificate in a chain file available to validate the trust chain of the peer x509 certificate when TLS mutual authentication is going on. All fine. Let’s suppose I have now on my network two servers and two clients and I want to make sure Client_A and Server_A succeed with TLS mutual authentication using their x509 certificates, as Client_B and Server_B should do. At the same time, I want to make sure TLS mutual authentication between Client_A and Server_B will not succeed. (Also valid between Client_B and Server_A).

How can I make the servers and clients in my network not only verifying the trust chain, but also respecting some kind of whitelist?

Maybe this is not feasible on TLS Layer, that is what I want to have clarified.

Answer 1

You can achieve this using any of the below 2 approaches-

1. Verifying client certificate at TLS layer: Create separate intermediate issuing CA for each client group. For example, for all the clients of Server_A, create a CA named Ca_Client_A. So your chain will look like rootCA -> Ca_Client_A -> client certificates. Import this Ca_Client_A in the trust-store of Server-A. Now Server_A will allow connections to the clients which has issuing ca Ca_Client_A. Similarly, you can create Ca_Client_B intermediate CA for serving client group B by server B.

2. Verifying client certificate at Application layer: For this, you need to write authentication logic in your application server where you need to put constraints such as allowed client certificate-serial numbers or CN name matching with keyword for successful authentication.

Hope that helps.

Answer 2

There isn't any way (that I know of) to do this at the TLS layer.

Most TLS libraries offer a callback option during the certificate exchange, and that would be the appropriate place to check the certificate against a list. Returning the library-specific version of failure/unacceptable will usually cause a TLS handshake failure, and no data will have been transmitted.

Answer 3

A lot of this depends on the server software you are using, not the TLS standard itself. Some software packages can be configured to trust certificates issued by a particular CA certificate. In your scenario, this would allow ANY certificate issued by your CA to connect to ANY server configured to trust your CA certificate. Other server software can be configured to trust a particular certificate, or certificates with a given subject (distinguished name of the certificate, subjectAltName, etc.) So, it depends on which particular "server" you're trying to connect to. Is this an Apache web server? IIS? Tomcat? The answer varies depending on what server platform we're talking about.