How to detect an incoming SSL (https) handshake (SSL wire format)?

Question

I'm writing a server which is accepting incoming TCP connections. Let's suppose the server has accepted a TCP connection, and has already received 16 (or so) bytes from the client. Knowing those 16 bytes how can the server detect whether the client wants to initiate an SSL handshake?

I've made an experiment, which showed that on my Linux system connecting to localhost (either 127.0.0.1 or AF_UNIX) via SSL makes the client send the following handshake (hexdump), followed by 16 seemingly random bytes:

8064010301004b0000001000003900003800003500001600001300000a07
00c000003300003200002f03008000000500000401008000001500001200
0009060040000014000011000008000006040080000003020080

How should the server probe these first few bytes, just to be able to determine whether the client is sending an SSL handshake? The probe must return true for all valid SSL handshakes, and it must return false with high probability for a message sent by the client which is not an SSL handshake. It is not allowed to use any libraries (like OpenSSL) for the probe. The probe must be a simple code (like a few dozen lines in C or Python).

Answer 1

The client always sends so called HelloClient message first. It can be in SSL 2 format or SSL 3.0 format (the same format as in TLS 1.0, 1.1 and 1.2).

And there is also possibility that SSL 3.0/TLS 1.0/1.1/1.2 clients send HelloClient with the older format (SSL 2), just with the higher version number in the data. So, detection of SSL 2 HelloClient is neccessary for newer clients too. (For example Java SSL implementation does so)

Let's say 'b' is your buffer. I tried to diagram the message format.

SSL 2

+-----------------+------+-------
| 2 byte header   | 0x01 | etc.
+-----------------|------+-------

• b[0] & 0x80 == 0x80 (it means most significant bit of b[0] is '1')

• ((b[0] & 0x7f) << 8 | b[1]) > 9 (It menas the low 7 bits of b[0] together with b[1] are length of data. You can have less in your buffer, so you cannot check them. But from the message format we know there are 3 field of 2 bytes (length fields), and at least one item in cipher list field (of size 3). So there should be at least 9 bytes (data length >= 9).

• b[2] must be 0x01 (message type "ClientHello")

SSL 3.0 or TLS 1.0, 1.1 and 1.2

+-------+------------------+------------------+--------+------
| 0x16  | 2 bytes version  |  2 bytes length  |  0x01  |  etc.
+-------+------------------+------------------+--------+------

• b[0] == 0x16 (message type "SSL handshake")

• b[1] should be 0x03 (currently newest major version, but who knows in future?)

• b[5] must be 0x01 (handshake protocol message "HelloClient")

For reference, you can see http://www.mozilla.org/projects/security/pki/nss/ssl/draft02.html and https://www.rfc-editor.org/rfc/rfc4346

Answer 2

I could figure this out based on the implementation of the ClientHello.parse method in http://tlslite.cvs.sourceforge.net/viewvc/tlslite/tlslite/tlslite/messages.py?view=markup

I am giving two solutions here in Python. IsSSlClientHandshakeSimple is a simple regexp, which can yield some false positives quite easily; IsSslClientHandshake is more complicated: it checks the consistency of lengths, and the range of some other numbers.

import re

def IsSslClientHandshakeSimple(buf):
  return bool(re.match(r'(?s)\A(?:\x80[\x0f-\xff]\x01[\x00-\x09][\x00-\x1f]'
                       r'[\x00-\x05].\x00.\x00.|'
                       r'\x16[\x2c-\xff]\x01\x00[\x00-\x05].'
                       r'[\x00-\x09][\x00-\x1f])', buf))

def IsSslClientHandshake(buf):
  if len(buf) < 2:  # Missing record header.
    return False
  if len(buf) < 2 + ord(buf[1]):  # Incomplete record body.
    return False
  # TODO(pts): Support two-byte lengths in buf[1].
  if ord(buf[0]) == 0x80:  # SSL v2.
    if ord(buf[1]) < 9:  # Message body too short.
      return False
    if ord(buf[2]) != 0x01:  # Not client_hello.
      return False
    if ord(buf[3]) > 9:  # Client major version too large. (Good: 0x03)
      return False
    if ord(buf[4]) > 31:  # Client minor version too large. (Good: 0x01)
      return False
    cipher_specs_size = ord(buf[5]) << 8 | ord(buf[6])
    session_id_size = ord(buf[7]) << 8 | ord(buf[8])
    random_size = ord(buf[9]) << 8 | ord(buf[10])
    if ord(buf[1]) < 9 + cipher_specs_size + session_id_size + random_size:
      return False
    if cipher_specs_size % 3 != 0:  # Cipher specs not a multiple of 3 bytes.
      return False
  elif ord(buf[0]) == 0x16:  # SSL v1.
    # TODO(pts): Test this.
    if ord(buf[1]) < 39:  # Message body too short.
      return False
    if ord(buf[2]) != 0x01:  # Not client_hello.
      return False
    head_size = ord(buf[3]) << 16 | ord(buf[4]) << 8 | ord(buf[5])
    if ord(buf[1]) < head_size + 4:  # Head doesn't fit in message body.
      return False
    if ord(buf[6]) > 9:  # Client major version too large. (Good: 0x03)
      return False
    if ord(buf[7]) > 31:  # Client minor version too large. (Good: 0x01)
      return False
    # random is at buf[8 : 40]
    session_id_size = ord(buf[40])
    i = 41 + session_id_size
    if ord(buf[1]) < i + 2:  # session_id + cipher_suites_size doesn't fit.
      return False
    cipher_specs_size = ord(buf[i]) << 8 | ord(buf[i + 1])
    if cipher_specs_size % 2 != 0:
      return False
    i += 2 + cipher_specs_size
    if ord(buf[1]) < i + 1: # cipher_specs + c..._methods_size doesn't fit.
      return False
    if ord(buf[1]) < i + 1 + ord(buf[i]): # compression_methods doesn't fit.
      return False
  else:  # Not SSL v1 or SSL v2.
    return False
return True