OpenSSL Custom Memory BIO for a sockets library

3
votes

I'm using a transport library that emulates Berkeley sockets API for working with it (complete with blocking and non-blocking modes). Needed to add TLS encryption to the data that I'm sending and can think of two ways:

  1. Custom BIO: Found code for the Socket BIO, so thinking of making a copy of it and replacing all berkeley socket calls with the functions for that library. However, I can't find much info about Custom BIOs online, so wary of pitfalls I'd run into without getting much help.

  2. Memory BIO: This approach has more followers and quite a few examples as well. Although, most warn that Memory BIO with non-blocking sockets is an order of magnitude more complicated than blocking route.

Regarding Memory BIO approach, The source code that I saw made sense to me but it was a simple echo client/server. The thing that's confusing me a lot is what to do when SSL_Read/SSL_Write return SSL_WANT_READ or SSL_WANT_WRITE. My understanding is that in case of Blocking Socket BIO, you just retry the call later, since underlying code will take care of things.

In case of Memory BIO + non-blocking sockets e.g. if SSL_Read returns SSL_WANT_WRITE, does that mean that my code should then read from output BIO (BIO_read) and send it to the socket and also don't allow any SSL_Read/SSL_Write until the original SSL_Read call succeeds? is it ok to allow SSL_Write during this time?

EDIT: I'll be using TLS 1.3 exclusively, just found out that it does Not support renegotiation. Does it mean that once connection is established I don't have to worry about WANT_WRITE on SSL_Read and WANT_READ on SSL_Write at all?

1
openssl

1 Answers

1
votes

"In case of Memory BIO + non-blocking sockets e.g. if SSL_Read returns SSL_WANT_WRITE, does that mean that my code should then read from output BIO (BIO_read) and send it to the socket" -- Yes. Take all the bytes in the output BIO and write them to the socket.

It's explained here why this can happen:

Any TLS/SSL I/O function can lead to either of SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE. In particular, SSL_read_ex(), SSL_read(), SSL_peek_ex(), or SSL_peek() may want to write data and SSL_write() or SSL_write_ex() may want to read data. This is mainly because TLS/SSL handshakes may occur at any time during the protocol (initiated by either the client or the server); SSL_read_ex(), SSL_read(), SSL_peek_ex(), SSL_peek(), SSL_write_ex(), and SSL_write() will handle any pending handshakes.

Once the SSL_read has failed in this manner, I don't think further attempt at SSL_read will succeed until you have written more bytes into the input bio.

Meanwhile you can make further attempts at SSL_write; just inspect the return code and SSL status to check whether the data was partially of fully transferred in.

I have some example code written on github