Auto-detecting if a TCP connection is using SSL / TLS?

3
votes

I have a C++ application that accepts plain and SSL / TLS encrypted connections. Currently it does this on 2 separate ports.

I would like to be able to use a single port and detect if the client is using TLS or not and respond accordingly.

Is it possible to "peek" into the socket connection without interfereing with the initial TLS handshake ?

I'm using OpenSSL to implement TLS.

2
c++socketstcpnetwork-programmingopenssl

2 Answers

3
votes

You just need to peek the first few bytes of a new incoming connection without consuming them (call recv() with the MSG_PEEK flag) and see if they look like an SSL ClientHello message as defined in RFC 2246. Then decide whether to continue as plaintext or via SSL.

2
votes

I don't believe there's an easy solution to that task.

Taken from IETF.

RFC 2246 The TLS Protocol Version 1.0 January 1999

receiving party may decide at its discretion whether to treat this as a fatal error or not. However, all messages which are transmitted with a level of fatal must be treated as fatal messages.

7.3. Handshake Protocol overview

The cryptographic parameters of the session state are produced by the TLS Handshake Protocol, which operates on top of the TLS Record Layer. When a TLS client and server first start communicating, they
agree on a protocol version, select cryptographic algorithms,
optionally authenticate each other, and use public-key encryption
techniques to generate shared secrets.

The TLS Handshake Protocol involves the following steps:

  • Exchange hello messages to agree on algorithms, exchange random values, and check for session resumption.

  • Exchange the necessary cryptographic parameters to allow the client and server to agree on a premaster secret.

  • Exchange certificates and cryptographic information to allow the client and server to authenticate themselves.

  • Generate a master secret from the premaster secret and exchanged random values.

  • Provide security parameters to the record layer.

  • Allow the client and server to verify that their peer has calculated the same security parameters and that the handshake occurred without tampering by an attacker.

Therefore you should sniff the handshake message, determine it's format and conclude to whether the client attempted to use TLS.

Another helpful link.

An SO old post.