Avoiding fragmented ClientHellos with OpenSSL (DTLS)

1
votes

I've been trying to write a DTLS server/client with OpenSSL, but I've run into this problem where DTLSv1_listen fails because the ClientHello message is fragmented (i've read that this is because DTLSv1_listen has to remain stateless to prevent DDoS attacks).

How would I avoid fragmented ClientHellos?

Note: The fragmented ClientHellos seem to only happen on mac. When I run my code on ubuntu it works without a problem but my mac build crashes because of the fragmented ClientHello.

UPDATE: It turns out that in addition to the fragmented ClientHellos, the timeout for poll was 0ms for some reason. After fixing that, i tried again.

On the client side, there was a

error:14102410:SSL routines:dtls1_read_bytes:sslv3 alert handshake failure

On the server side (fragmented client hello still there):

error:1415E191:SSL routines:DTLSv1_listen:fragmented client hello
error:14209134:SSL routines:tls_early_post_process_client_hello:cookie mismatch

I'm a wireshark noob but I made this capture (hope it helps): https://drive.google.com/file/d/1mieHlZa8zG7Yvd8M615X6AIpdOet8uXv/view?usp=sharing

1
c++openssldtls
Can you capture a wireshark trace of a failing connection attempt. What is the size of the fragmented ClientHello, and what is the MTU for the connection? – Matt Caswell
This seems a wire problem. – Manuel

1 Answers

0
votes

In many cases, openssl comes with many cipher suites and other algorithms, which are put into the HELLO_CLIENT. If that is too much for your message size, try to configure openssl to use a subset of the ciphersuites (maybe you know, what the server supports). This helps in many cases to fit the CLIENT_HELLO into one message.