when server written in openssl is'nt calling SSL_accept, client's SSL_connect blocks forever. There are some timeout functions in openssl -SSL_CTX_set_timeout , SSL_SESSION_set_timeout but these have no effect on SSL_connect.
Is there really no way of setting timeout for SSL_connect when e.g. ssl server is buggy and goes into loop before doing SSL handshake?
The OpenSSL Library gives you the maximum flexibility in terms of handling socket related issues. The SSL_connect blocks in your case because you must be using it with a blocking socket. Please use it with a non-blocking socket, in which case it will return with a -1. If you call SSL_get_error function which will give you SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE error depending on when the tcp recv or send is unable to complete the operation respectively.
When SSL_ERROR_WANT_WRITE/SSL_ERROR_WANT_READ is obtained, you must call select function by passing the socket to appropriate fd_set and a timeout. If the select times out, you can consider your SSL_connect to have timed out.
Note: The SSL_SESSION_set_timeout is used for setting session timeout values which are linked to SSL resumption. They have nothing to do with timing out a connection.
The below links should help you (especially the second link, Section 6 which talks about Multiplexed I/O):
