2
votes

I'm working with Windows Server 2012, Erlang 19.2, and RabbitMq 3.6.6. I'm having trouble configuring the connection between endpoints using TLS. I've tried every answer on SO, as well as all the RabbitMq docs here and here. Not sure what we're doing wrong.

In the troubleshooting link here all tests pass except the "Attempt SSL connection to broker" piece. This is where the problem lies and I'm not sure why.

When I go through the documentation on troubleshooting to see if you can get a peer connection over SSL on port 8443, it works fine. Then trying to connect to the broker on port 5671 fails, saying bad handshake.

Switching the RabbitMq config file to 8443 does nothing, other than make the peer to peer work on 5671 and fail on 8443.

My config file:

[
  {rabbit, [
     {ssl_listeners, [5671]},
     {ssl_options, [{cacertfile,"C:\\rabbitcerts\\testca\\cacert.pem"},
                    {certfile,"C:\\rabbitcerts\\server\\cert.pem"},
                    {keyfile,"C:\\rabbitcerts\\server\\key.pem"},
                    {depth, 2},
                    {verify,verify_peer},
                    {fail_if_no_peer_cert,false}]}
   ]}
].

Running this command:

c:\rabbitcerts>openssl s_client -connect localhost:5671 -cert client/cert.pem -key client/key.pem -CAfile testca/cacert.pem

Produces this error:

Loading 'screen' into random state - done
CONNECTED(000001BC)
write:errno=10054

And in the log file:

=INFO REPORT==== 19-Jan-2017::16:42:50 ===
Memory limit set to 716MB of 1791MB total.

=INFO REPORT==== 19-Jan-2017::16:42:50 ===
Disk free limit set to 50MB

=INFO REPORT==== 19-Jan-2017::16:42:50 ===
Limiting to approx 8092 file handles (7280 sockets)

=INFO REPORT==== 19-Jan-2017::16:42:50 ===
FHC read buffering:  OFF
FHC write buffering: ON

=INFO REPORT==== 19-Jan-2017::16:42:50 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Starting rabbit_node_monitor

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Management plugin: using rates mode 'basic'

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
started TCP Listener on [::]:5672

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
started TCP Listener on 0.0.0.0:5672

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
started SSL Listener on [::]:5671

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
started SSL Listener on 0.0.0.0:5671

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Management plugin started. Port: 15672

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics event collector started.

...

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics database started.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_queue_stats_fine_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_queue_stats_deliver_get with interval 5000.

...

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_queue_exchange_stats_fine_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_vhost_stats_deliver_get with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_vhost_stats_fine_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_vhost_stats_queue_msg_rates with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_vhost_stats_queue_msg_counts with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_vhost_stats_coarse_conn_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_channel_queue_stats_deliver_get with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_channel_queue_stats_fine_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_channel_queue_stats_queue_msg_counts with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_channel_stats_deliver_get with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_channel_stats_fine_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_channel_stats_queue_msg_counts with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_channel_stats_process_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_channel_exchange_stats_deliver_get with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_channel_exchange_stats_fine_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_exchange_stats_fine_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table aggr_node_stats_coarse_node_stats with interval 5000.

...

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Statistics garbage collector started for table connection_stats with interval 5000.

=INFO REPORT==== 19-Jan-2017::16:42:51 ===
Server startup complete; 6 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * webmachine
 * mochiweb
 * rabbitmq_management_agent
 * amqp_client

=ERROR REPORT==== 19-Jan-2017::16:54:39 ===
SSL: hello: tls_handshake.erl:202:Fatal error: handshake failure - handshake_decode_error

What on Earth am I missing?

I've reached out to my network admin to see if there is a configuration on the server that we might be missing, per this answer on SO, but I'd like to hear from others, as I'm sure I can't be the only one encountering any issues...

UPDATE

It appears I'm getting closer using the new command from @jww.

openssl s_client -connect mymachine:5671 -tls1 -servername mymachine

Output:

Loading 'screen' into random state - done
CONNECTED(000001BC)
depth=1 /CN=MyTestCA
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/CN=$(hostname)/O=server
   i:/CN=MyTestCA
 1 s:/CN=MyTestCA
   i:/CN=MyTestCA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIC5DCCAcygAwIBAgIBATANBgkqhkiG9w0BAQsFADATMREwDwYDVQQDEwhNeVRl
c3RDQTAeFw0xNzAxMTkxNjA1NDhaFw0xODAxMTkxNjA1NDhaMCcxFDASBgNVBAMU
CyQoaG9zdG5hbWUpMQ8wDQYDVQQKEwZzZXJ2ZXIwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC1WnL4V7VWwi9EytZT1UTR3ixQcXwCSWDe3aS8yk1KFadL
1ZPBgj3ZYDs/NwDX/KJ/d31yCgpwl/ZS6lWjn2Ect7BfHwKHd98L5SVl9Na2TPUP
73kLdITDYvJbACoQu+JT60CNPBXsTPww2L2OpFYUhDSXGwV721Y5rcaU9a2VPzjp
N0puT8qdxMmOz7Zp2WAjmkmSRpbOz2Z3/BbVI9zPMYLenmOeoLDOpM2vGqeLRSy1
ruBd7Rw3gFKvYN/flXZyfZkqrY5FOju6okp6n9KvnibnmgATS1OuSmADFS78x0Zz
XM7Cep23b4Ix+ckB4PzpAwRKsiWv534veN1lK42hAgMBAAGjLzAtMAkGA1UdEwQC
MAAwCwYDVR0PBAQDAgUgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMA0GCSqGSIb3DQEB
CwUAA4IBAQBolBD+sy7H1SdtgGsS45eYp1zSEPlOEZLZhmCsN4zN4rG0Qo6SGEvd
cODk3hIWfglgb50oouGGebE84ReTSLQvFp9eGoIokB8azy2l25weZPvyPjjkdBiF
/XI3Wn/oJaRX9t2nnMZjQE14W22KqwGewMh0PywdLcjV6llqmFzZAQv6GTIvyOZw
QqCZjanYXGtyi3QSK6D1MxBaDW7hg4/WaUkNEhKVEQ6Vm3EvnvGVD6XZVP7RM7Iy
oN7wXuGlasoBx7Zs5sJh1/uNYyN2QHYKu8z5tLgXACzA9phNLeOGaimxIZIUAjnJ
IY08bwLeo/hbDKNA3hvyQlgSpy7t2U4o
-----END CERTIFICATE-----
subject=/CN=$(hostname)/O=server
issuer=/CN=MyTestCA
---
Acceptable client certificate CA names
/CN=MyTestCA
---
SSL handshake has read 1659 bytes and written 453 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: 0E00F18E516DBD5C7EE7F7FE070BDC09FBE3B731FA8D1DF2ECD75E455BB8A6EF
    Session-ID-ctx:
    Master-Key: 61F018A5B629EE6015F88B076AEA8765E153A8CCB2241766DFD0BCC369DC703C9BF42249E47C93EEA318899615732390
    Key-Arg   : None
    Start Time: 1484872012
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
closed
2
is this a development or production environment?Alex Buyny
@AlexBuyny dev.ragerory
@jww I posted the s_client command I am using. In a dev environment, and with a broker on the same machine, it should work fine, according to every other document - that article is under the assumption it's over the net.ragerory
@ragerory - The command you are using is probably broken. Please post the output of s_client -connect example.com:443 -tls1 -servername example.com. Use the server name and port. If it does not work with TLS 1.0 (-tls1), then try TLS 1.2 (-tls1_2).jww
Actually, when I took my original command and added -tls1 to it, it worked correctly and gave me a Verify code: 0 (ok) - so it appears it was using the wrong type on the broker. Thanks for getting me where I needed to be @jwwragerory

2 Answers

2
votes

In this particular case, everything was setup correctly. However, it seems when creating a peer connection in the RabbitMq Console for troubleshooting, it creates a connection over a different protocol than when trying to connect to the broker.

So, where this didn't work:

openssl s_client -connect localhost:5671 -cert client/cert.pem -key client/key.pem -CAfile testca/cacert.pem

I added -tls1 to the arguments, per @jww's other recommendation, and that was all I needed to create the secure connection.

openssl s_client -connect localhost:5671 -tls1 -cert client/cert.pem -key client/key.pem -CAfile testca/cacert.pem

Resulting in a Verify code: (ok).

Loading 'screen' into random state - done
CONNECTED(000001BC)
depth=1 /CN=MyTestCA
verify return:1
depth=0 /CN=$(hostname)/O=server
verify return:1
---
Certificate chain
 0 s:/CN=$(hostname)/O=server
   i:/CN=MyTestCA
 1 s:/CN=MyTestCA
   i:/CN=MyTestCA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIC5DCCAcygAwIBAgIBATANBgkqhkiG9w0BAQsFADATMREwDwYDVQQDEwhNeVRl
c3RDQTAeFw0xNzAxMTkxNjA1NDhaFw0xODAxMTkxNjA1NDhaMCcxFDASBgNVBAMU
CyQoaG9zdG5hbWUpMQ8wDQYDVQQKEwZzZXJ2ZXIwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC1WnL4V7VWwi9EytZT1UTR3ixQcXwCSWDe3aS8yk1KFadL
1ZPBgj3ZYDs/NwDX/KJ/d31yCgpwl/ZS6lWjn2Ect7BfHwKHd98L5SVl9Na2TPUP
73kLdITDYvJbACoQu+JT60CNPBXsTPww2L2OpFYUhDSXGwV721Y5rcaU9a2VPzjp
N0puT8qdxMmOz7Zp2WAjmkmSRpbOz2Z3/BbVI9zPMYLenmOeoLDOpM2vGqeLRSy1
ruBd7Rw3gFKvYN/flXZyfZkqrY5FOju6okp6n9KvnibnmgATS1OuSmADFS78x0Zz
XM7Cep23b4Ix+ckB4PzpAwRKsiWv534veN1lK42hAgMBAAGjLzAtMAkGA1UdEwQC
MAAwCwYDVR0PBAQDAgUgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMA0GCSqGSIb3DQEB
CwUAA4IBAQBolBD+sy7H1SdtgGsS45eYp1zSEPlOEZLZhmCsN4zN4rG0Qo6SGEvd
cODk3hIWfglgb50oouGGebE84ReTSLQvFp9eGoIokB8azy2l25weZPvyPjjkdBiF
/XI3Wn/oJaRX9t2nnMZjQE14W22KqwGewMh0PywdLcjV6llqmFzZAQv6GTIvyOZw
QqCZjanYXGtyi3QSK6D1MxBaDW7hg4/WaUkNEhKVEQ6Vm3EvnvGVD6XZVP7RM7Iy
oN7wXuGlasoBx7Zs5sJh1/uNYyN2QHYKu8z5tLgXACzA9phNLeOGaimxIZIUAjnJ
IY08bwLeo/hbDKNA3hvyQlgSpy7t2U4o
-----END CERTIFICATE-----
subject=/CN=$(hostname)/O=server
issuer=/CN=MyTestCA
---
Acceptable client certificate CA names
/CN=MyTestCA
---
SSL handshake has read 1659 bytes and written 2163 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: 56CC3AB350BF91DB4CD2A89F62FD60322E553628C381E11B179BD9C8D22184BF
    Session-ID-ctx:
    Master-Key: 6FB8A241FD0A5C3ECCBE88DE4C36C412CBE5E8D58DAAB209D24438F72CCA7F9332511A277EBC0919775490057F46CCC7
    Key-Arg   : None
    Start Time: 1484921846
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
0
votes

I had "connection reset by peer" error recently when setting up dev rabbit. things to try:

  1. Install Erlang 18.2 instead of 19.2. I had no luck at all with 19.X versions when setting up TLS. I had strange errors and nothing being logged by RabbitMQ.
  2. when trying Erlang 18.2 look at your_instance-sasl.log, TLS problems were logged there for me with enough detail so I was able to resolve those problems.

I myself have RabbitMQ server installed on Centos7 and client on Windows. For dev environment I used tls-gen to generate certificates. Was very easy.