How to detect an incoming SSL (https) handshake (SSL wire format)?

Question

I\'m writing a server which is accepting incoming TCP connections. Let\'s suppose the server has accepted a TCP connection, and has already received 16 (or so) bytes from th

Answer 1

I could figure this out based on the implementation of the ClientHello.parse method in http://tlslite.cvs.sourceforge.net/viewvc/tlslite/tlslite/tlslite/messages.py?view=markup

I am giving two solutions here in Python. IsSSlClientHandshakeSimple is a simple regexp, which can yield some false positives quite easily; IsSslClientHandshake is more complicated: it checks the consistency of lengths, and the range of some other numbers.

import re

def IsSslClientHandshakeSimple(buf):
  return bool(re.match(r'(?s)\A(?:\x80[\x0f-\xff]\x01[\x00-\x09][\x00-\x1f]'
                       r'[\x00-\x05].\x00.\x00.|'
                       r'\x16[\x2c-\xff]\x01\x00[\x00-\x05].'
                       r'[\x00-\x09][\x00-\x1f])', buf))

def IsSslClientHandshake(buf):
  if len(buf) < 2:  # Missing record header.
    return False
  if len(buf) < 2 + ord(buf[1]):  # Incomplete record body.
    return False
  # TODO(pts): Support two-byte lengths in buf[1].
  if ord(buf[0]) == 0x80:  # SSL v2.
    if ord(buf[1]) < 9:  # Message body too short.
      return False
    if ord(buf[2]) != 0x01:  # Not client_hello.
      return False
    if ord(buf[3]) > 9:  # Client major version too large. (Good: 0x03)
      return False
    if ord(buf[4]) > 31:  # Client minor version too large. (Good: 0x01)
      return False
    cipher_specs_size = ord(buf[5]) << 8 | ord(buf[6])
    session_id_size = ord(buf[7]) << 8 | ord(buf[8])
    random_size = ord(buf[9]) << 8 | ord(buf[10])
    if ord(buf[1]) < 9 + cipher_specs_size + session_id_size + random_size:
      return False
    if cipher_specs_size % 3 != 0:  # Cipher specs not a multiple of 3 bytes.
      return False
  elif ord(buf[0]) == 0x16:  # SSL v1.
    # TODO(pts): Test this.
    if ord(buf[1]) < 39:  # Message body too short.
      return False
    if ord(buf[2]) != 0x01:  # Not client_hello.
      return False
    head_size = ord(buf[3]) << 16 | ord(buf[4]) << 8 | ord(buf[5])
    if ord(buf[1]) < head_size + 4:  # Head doesn't fit in message body.
      return False
    if ord(buf[6]) > 9:  # Client major version too large. (Good: 0x03)
      return False
    if ord(buf[7]) > 31:  # Client minor version too large. (Good: 0x01)
      return False
    # random is at buf[8 : 40]
    session_id_size = ord(buf[40])
    i = 41 + session_id_size
    if ord(buf[1]) < i + 2:  # session_id + cipher_suites_size doesn't fit.
      return False
    cipher_specs_size = ord(buf[i]) << 8 | ord(buf[i + 1])
    if cipher_specs_size % 2 != 0:
      return False
    i += 2 + cipher_specs_size
    if ord(buf[1]) < i + 1: # cipher_specs + c..._methods_size doesn't fit.
      return False
    if ord(buf[1]) < i + 1 + ord(buf[i]): # compression_methods doesn't fit.
      return False
  else:  # Not SSL v1 or SSL v2.
    return False
return True

Answer 2

The client always sends so called HelloClient message first. It can be in SSL 2 format or SSL 3.0 format (the same format as in TLS 1.0, 1.1 and 1.2).

And there is also possibility that SSL 3.0/TLS 1.0/1.1/1.2 clients send HelloClient with the older format (SSL 2), just with the higher version number in the data. So, detection of SSL 2 HelloClient is neccessary for newer clients too. (For example Java SSL implementation does so)

Let's say 'b' is your buffer. I tried to diagram the message format.

SSL 2

+-----------------+------+-------
| 2 byte header   | 0x01 | etc.
+-----------------|------+-------

• b[0] & 0x80 == 0x80 (it means most significant bit of b[0] is '1')

• ((b[0] & 0x7f) << 8 | b[1]) > 9 (It menas the low 7 bits of b[0] together with b[1] are length of data. You can have less in your buffer, so you cannot check them. But from the message format we know there are 3 field of 2 bytes (length fields), and at least one item in cipher list field (of size 3). So there should be at least 9 bytes (data length >= 9).

• b[2] must be 0x01 (message type "ClientHello")

SSL 3.0 or TLS 1.0, 1.1 and 1.2

+-------+------------------+------------------+--------+------
| 0x16  | 2 bytes version  |  2 bytes length  |  0x01  |  etc.
+-------+------------------+------------------+--------+------

• b[0] == 0x16 (message type "SSL handshake")

• b[1] should be 0x03 (currently newest major version, but who knows in future?)

• b[5] must be 0x01 (handshake protocol message "HelloClient")

For reference, you can see http://www.mozilla.org/projects/security/pki/nss/ssl/draft02.html and http://tools.ietf.org/html/rfc4346