OpenSSL ssl_accept() error 5

Question

I have scoured Stack Overflow and the internet, but I have been unable to locate an answer to why ssl_accept() keeps returning:

[DEBUG] SSL_accept() : Failed with return 0
[DEBUG]     SSL_get_error() returned : 5
[DEBUG]     Error string : error:00000005:lib(0):func(0):DH lib
[DEBUG]     WSAGetLastError() returned : 0
[DEBUG]     GetLastError() returned : 0
[DEBUG]     ERR_get_error() returned : 0

Edit: Out of interest ssl_accept() returns 0, defined as (accordingly to the scant and unhelpful OpenSSL documentation): "The TLS/SSL handshake was not successful but was shut down controlled and by the specifications of the TLS/SSL protocol. Call SSL_get_error() with the return value ret to find out the reason."

Below is the snippet of the server side, could I be barking up the wrong tree and this issue be caused by client code?

    client = accept( server, (sockaddr*) &clientsockaddrin, &len );

    SSL* ssl = SSL_new( ctx );

    SSL_set_fd( ssl, client );

    std::cout << "+--------------------------------------------------+"
              << std::endl;

    int r = SSL_accept( ssl );

    if ( r != 1 ) 
    {
        int err_SSL_get_error = SSL_get_error( ssl, r);
        int err_GetLastError = GetLastError();
        int err_WSAGetLastError = WSAGetLastError();
        int err_ERR_get_error = ERR_get_error();

        std::cout << "[DEBUG] SSL_accept() : Failed with return " 
                  << r << std::endl;
        std::cout << "[DEBUG]     SSL_get_error() returned : "
                  << err_SSL_get_error << std::endl;
        std::cout << "[DEBUG]     Error string : "
                  << ERR_error_string( err_SSL_get_error, NULL ) 
                  << std::endl;
        std::cout << "[DEBUG]     WSAGetLastError() returned : "
                  << err_WSAGetLastError << std::endl;
        std::cout << "[DEBUG]     GetLastError() returned : "
                  << err_GetLastError << std::endl;
        std::cout << "[DEBUG]     ERR_get_error() returned : "
                  << err_ERR_get_error << std::endl;
        std::cout << "+--------------------------------------------------+"
                  << std::endl;
        break;
    }

Appreciate your assistance as this is driving me mad :(

This is what I am not sure of, I can't find any definitive error codes list, without going through the OpenSSL source code I guess? — Gemma Morriss
Did client = accept(...) fail? Its not clear if you are performing error checking. The Vista and above (or even XP SP 2) firewall will deny access by default unless you've placed a exception in the firewall rules. — jww
WSAGetLastError function will help you with the error codes. Also, you should not delay the call to WSAGetLastError (or GetLastError). Call it immediately after SSL_accept. If SSL_accept succeeds, then simply ignore the return value. — jww
Yeah, the only refs I found on the internet seemed to be because of failed connections on this one. — Maarten Bodewes

Steffen Ullrich Steffen Ullrich · Accepted Answer · 2014-05-07T03:17:39

[DEBUG] Error string : error:00000005:lib(0):func(0):DH lib

The error happened during the Diffie-Hellman Key Exchange, e.g. where the peers tried to generate the keys for the connection. There might be several reasons for this, like invalid DH parameters given on the server side. With the your current code it is hard to see where the error actually is, but I guess is somewhere in setting up your ctx, so maybe should show the relevant parts of the code.

OpenSSL ssl_accept() error 5

2 Answers