C++ OpenSSL Fails to perform handshake when accepting in non-blocking mode. What is the proper way?

Question

I'm trying to implement OpenSSL into my application which uses raw C sockets and the only issue I'm having is the SSL_accept / SSL_connect part of the code which starts the KeyExchange phase but does not seem to complete it on the serverside.

I've had a look at countless websites and Q&A's here on StackOverflow to get myself through the OpenSSL API since this is basically the first time I'm attempting to implement SSL into an application but the only thing I could not find yet was how to properly manage failed handshakes.

Basically, running process A which serves as a server will listen for incoming connections. Once I run process B, which acts as a client, it will successfully connect to process A but SSL_accept (on the server) fails with error code -2 SSL_ERROR_WANT_READ.

According to openssl handshake failed, the problem is "easily" worked around by calling SSL_accept within a loop until it finally returns 1 (It successfully connects and completes the handshake). However, I do not believe that this is the proper way of doing things as it looks like a dirty trick. The reason for why I believe it is a dirty trick is because I tried to run a small application I found on https://www.cs.utah.edu/~swalton/listings/articles/ (ssl_client and ssl_server) and magically, everything works just fine. There are no multiple calls to SSL_accept and the handshake is completed right away.

Here's some code where I'm accepting the SSL connection on the server:

if (SSL_accept(conn.ssl) == -1)
            {
                fprintf(stderr, "Connection failed.\n");
                fprintf(stderr, "SSL State: %s [%d]\n", SSL_state_string_long(conn.ssl), SSL_state(conn.ssl));
                ERR_print_errors_fp(stderr);
                PrintSSLError(conn.ssl, -1, "SSL_accept");
                return -1;
            }
            else
            {
                fprintf(stderr, "Connection accepted.\n");
                fprintf(stderr, "Server -> Client handshake completed");
            }

This is the output of PrintSSLError:

SSL State: SSLv3 read client hello B [8465]
[DEBUG] SSL_accept : Failed with return -1
[DEBUG]     SSL_get_error() returned : 2
[DEBUG]     Error string : error:00000002:lib(0):func(0):system lib
[DEBUG]     ERR_get_error() returned : 0
[DEBUG]     errno returned : Resource temporarily unavailable

And here's the client side snippet which connects to the server:

if (SSL_connect(conn.ssl) == -1)
            {
                fprintf(stderr, "Connection failed.\n");
                ERR_print_errors_fp(stderr);
                PrintSSLError(conn.ssl, -1, "SSL_connect");
                return -1;
            }
            else
            {
                fprintf(stderr, "Connection established.\n");
                fprintf(stderr, "Client -> Server handshake completed");
                PrintSSLInfo(conn.ssl);
            }

The connection is successfully enstablished client-side (SSL_connect does not return -1) and PrintSSLInfo outputs:

Connection established.
Cipher: DHE-RSA-AES256-GCM-SHA384
SSL State: SSL negotiation finished successfully [3]

And this is how I wrap the C Socket into SSL:

SSLConnection conn;
        conn.fd = fd;
        conn.ctx = sslContext;

        conn.ssl = SSL_new(conn.ctx);
        SSL_set_fd(conn.ssl, conn.fd);

The code snippet here resides within a function that takes a file-descriptor of the accepted incoming connection on the raw socket and the SSL Context to use.

To initialize the SSL Contexts I use TLSv1_2_server_method() and TLSv1_2_client_method(). Yes, I know that this will prevent clients from connecting if they do not support TLS 1.2 but this is exactly what I want. Whoever connects to my application will have to do it through my client anyway.

Either way, what am I doing wrong? I'd like to avoid loops in the authentication phase to avoid possible hang ups/slow downs of the application due to unexpected infinite loops since OpenSSL does not specify how many attempts it might take.

The workaround that worked, but that I'd like to avoid, is this:

while ((accept = SSL_accept(conn.ssl)) != 1)

And inside the while loop I check for the return code stored inside accept.

Things I've tried to workaround the SSL_ERROR_WANT_READ error:

1. Added usleep(50) inside the while loop (still takes several cycles to complete)
2. Added SSL_do_handshake(conn.ssl) after SSL_connect and SSL_accept (didn't change anything on the end-result)
3. Had a look at the code shown on roxlu.com (search on Google for "Using OpenSSL with memory BIOs - Roxlu") to guide me through the handshaking phase but since I'm new to this, and I don't directly use BIOs in my code but simply wrap my native C sockets into SSL, it was kind of confusing. I'm also unable to re-write the Networking part of the application as it'd would be too much work for me right now.

I've done some tests with the openssl command-line as well to troubleshoot the issue but it gives no error. The handshake appears to be successful as no errors such as:

24069864:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:656

appear. Here's the whole output of the command

openssl s_client -connect IP:Port -tls1_2 -prexit -msg

http://pastebin.com/9u1bfuf4

Things to note: 1. I'm using the latest OpenSSL version 1.0.2h 2. Application runs on a Unix system 3. Using self-signed certificates to encrypt the network traffic

Thanks everyone who's going to help me out.

Edit: I forgot to mention that the sockets are in non-blocking mode since the application serves multiple clients in one-go. Though, client-side they are in blocking mode.

Edit2: Leaving this here for future reference: jmarshall.com/stuff/handling-nbio-errors-in-openssl.html

Answer 1

You have clarified that the socket question is non-blocking.

Well, that's your answer. Obviously, when the socket is in a non-blocking mode, the handshake cannot be immediately completed. The handshake involves an exchange of protocol packets between the client and the server, with each one having to wait to receive the response from its peer. This works fine when the socket is in its default blocking mode. The library simply read()s and write()s, which blocks and waits until the message gets succesfully read or written. This obviously can't happen when the socket is in the non-blocking mode. Either the read() or write() immediately succeeds, or fails, if there's nothing to read or if the socket's output buffer is full.

The manual pages for SSL_accept() and SSL-connect() explain the procedure you must implement to execute the SSL handshake when the underlying socket is in a non-blocking mode. Rather than repeating the whole thing here, you should read the manual pages yourself. The capsule summary is to use SSL_get_error() to determine if the handshake actually failed, or if the library wants to read or write to/from the socket; and in that eventuality call poll() or select(), accordingly, then call SSL_accept() and SSL_connect() again.

Any other approach, like sprinkling silly sleep() calls, here and there, will result in an unreliable house of cards, that will fail randomly.