Java8 Secure WebSocket Handshake Problems (Tyrus and Jetty)

Question

I'm attempting to implement a WebSocket Client in an application that supports secure transmissions through SSL. The application already supports standard SSL connections over HTTP, by implementing custom Key and Trust managers (these custom implementations are in place to prompt the user for a certificate when needed).

I'm having trouble getting a secure connection to our remote WebSocket endpoint. The failure is occurring during the handshake. I've tried two different implementations of the WebSocket API (both Tyrus and Jetty), and both fail in the same way, which, of course, leads me to point to our SSL implementation.

As I mentioned, the failure is occurring during the handshake. It seems that the connection cannot figure out that there are client certificates that are signed by the supported authorities returned from the server. I'm stumped to figure out if I haven't supplied the client certificates to the WebSocket API correctly, or if our custom Key/Trust managers are even getting used.

Here's a dump of the SSL Debug logs:

*** CertificateRequest
Cert Types: RSA, DSS
Cert Authorities:
(list of about 15 cert authorities supported by the server)
*** ServerHelloDone
Warning: no suitable certificate found - continuing without client authentication
*** CertificateChain
<empty>
***

I've set breakpoints in our TrustManager implementation, to determine if they are ever getting called, and it seems that they are not being called at this point.

I've been attempting to debug this for a few days now, and am running out of things to try.

Any suggestions would be greatly appreciated!

Here's a snippet of the Jetty Code:

SSLContext context = SSLContext.getInstance("TLS");
// getKeyManagers / getTrustManagers retrieves an 
// array containing the custom key and trust manager
// instances:
KeyManager[] km = getKeyManagers();
TrustManager[] tm = getTrustManagers();
context.init(km, tm, null);

SslContextFactory contextFactory = new SslContextFactory();
contextFactory.setContext(context);

WebSocketClient client = new WebSocketClient(contextFactory);
SimpleEchoClient echoClient = new SimpleEchoClient();

try { 
    client.start();
    ClientUpgradeRequest request = new ClientUpgradeRequest();
    Future<Session> connection = client.connect(echoClient, uri, request);

    Session session = connection.get();

    // if everything works, do stuff here

    session.close();
    client.destroy();
} catch (Exception e) {
    LOG.error(e);
}

Unclear what side those ssl logs are from, client? server? what's the server? how is the server setup/configured? does the server work with other ssl clients (eg: openssl s_client)? — Joakim Erdfelt
Those logs are from the client side. We also maintain a web (angular) version of the application, and this server endpoint does work with that implementation. — Robert K
If you skip your custom TrustManager and KeyManager and simply use new SslContextFactory(true); does it work? — Joakim Erdfelt
After client.start(); what is the value of contextFactory.getState()? — Joakim Erdfelt
After client.start(), the value of contextFactory.getState() is STARTED. Trying it without my custom managers and new SslContextFactory(true) results in the same behavior. — Robert K

nagoor hussain nagoor hussain · Accepted Answer · 2017-06-27T16:28:38

can you try with rejectUnAuthorized:false so that your certificates for which your browser is unable to authorize will skip the authorization.

var ws = new WebSocket('wss://localhost:xxxx', {
  protocolVersion: 8,
  origin: 'https://localhost:xxxx',
  rejectUnauthorized: false
});

Java8 Secure WebSocket Handshake Problems (Tyrus and Jetty)

1 Answers