Apache couchDB CA signed certificate issues

Question

I am running an Apache CouchDB instance (version 1.3.0) on an Ubuntu 12.10 server in the cloud (AWS). I am trying to get SSL working on my couchDB instance.

The basic SSL setup is very easy. I have placed my certificate and key in a directory and uncommented the following lines in my local.ini file

httpsd = {couch_httpd, start_link, [https]}
cert_file = /usr/local/etc/couchdb/certs/mycouchdbserver_cert.pem
key_file = /usr/local/etc/couchdb/certs/mycouchdbserver_key.pem

I have also made sure that the ownership on these files is correct.

This works fine, the couchDB server starts up, you can navigate to https://mycouchdbserver.com/_utils/ without a problem.

Testing using openssl

openssl s_client -showcerts -connect mycouchdbserver.com:443

Gives the correct result for standard SSL configuration

When testing the setup on the DigiCert website (the company the SSL certs were bought through - test link: http://www.digicert.com/help/) I get the following error:

The server is not sending the required intermediate certificate.

When purchasing the SSL certificate I obtained an intermediate certificate from DigiCert and have downloaded the root cert for DigiCert as well.

In the local.ini config file for couchDB you can use these with the following configuration fields:

verify_ssl_certificates = true
cacert_file = xxxx

My problem is that I cant get this to work and have tried every possible combination to get this to work. Here is what I have tried:

1. Tried setting cacert_file to the intermediate cert from DigiCert
2. Tried setting cacert_file to the root certificate in /etc/ssl/certs
3. Tried adding the root cert from DigiCert website to /usr/shared/ca-certs/ and then running dpkg-reconfigure ca-certificates to install a new root certificate and setting cacert_file to that new pem encoded certificate in /etc/ssl/certs
4. Tried combining the cert and intermediate cert in one file used for cert_file
5. Tried combining the cert, intermediate cert and root cert into 1 pem file used for cert_file

All of the above throws errors in the couchDB log. Some give a mass amount of output in the errors logs but using number 3, I get

=ERROR REPORT==== 11-Jun-2013::11:35:30 ===
SSL: hello: ssl_handshake.erl:252:Fatal error: internal error

And testing with openssl I get

CONNECTED(00000003)
16871:error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal    error:s3_pkt.c:1099:SSL alert number 80
16871:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:

Does anyone have any idea on how to use the verify_ssl_certificates, the root certificate and the intermediate certificate correctly with couchDB

I have read all documentation online and nothing has helped

Thanks in Advance Andrew

Answer 1

For anyone who is interested this is how we finally solved the problem:

Seems that we couldn't get couchDB to work properly with our intermediate certificate.

Since we are running our couchDB server on an AWS EC2 instance, I just created an ELB (Elastic Load Balancer) instance and uploaded my certificates to the ELB, then added the EC2 instance under my load balancer and rerouted my DNS to the load balancer (using Route53 here too).

I then turned off SSL completely on couchDB and handed the SSL handshake to the load balancer which supports the use of an intermediate certificate.

This does mean that the comms between the ELB and couchDB is insecure but for us that is fine.

This also means that we now can add in more couchDB servers under the ELB for scalability so 2 birds 1 stone solution.

You can do this same solution with Nginx as well but adding and managing an ELB is easy and stable so we went with the ELB solution.

Answer 2

There's a few things that CouchDB is sensitive to. One issue is namecheap's reissue form. If you use namecheap to process your CSR, you will have issues. For example, I bought a RapidSSL cert through Namecheap, and in order to reissue it properly I had to go directly to GeoTrust to get a working cert: https://products.geotrust.com/orders/orderinformation/authentication.do

To create the SSL certs, I did the following:

$ openssl genrsa -des3 -out server.pem 2048

A passphrase was used. It must be used for the other commands do don't forget it.

Create the cert request:

$ openssl req -new -key server.pem -out server.csr

Only answer the following:

• Country
• State
• Locality
• Organization
• Common name

All other questions were left blank.

This command creates the unencrypted private key that CouchDB will use:

$ openssl rsa -in server.pem -out server.key

When asked for the csr, past the contents of the server.csr file into the text box.

After a bunch of emails, you will eventually get the cert. Another issue is how CouchDB handles the chained cert. Don't worry about creating a chained certificate; ignore the intermediate crt's, you only need the specific cert for your domain for couchdb to work properly.

Modify the following lines in the CouchDB local.ini file:

[daemons]
; enable SSL support by uncommenting the following line and supply the PEM's below.
; the default ssl port CouchDB listens on is 6984
httpsd = {couch_httpd, start_link, [https]}

[ssl]
cert_file = /path/to/ssl/cert/server.crt
key_file = /path/to/ssl/cert/server.key

I'm not sure if this will make a difference, but make sure the permissions on the SSL certs are set to 600. Also make sure to chown the certs by the CouchDB process user:

# in the ssl cert directory
sudo chmod 600 ./*
sudo chown couchdb:couchdb ./*

And restart your server:

sudo /etc/init.d/couchdb restart

Answer 3

Summary: You'll need CouchDB 1.6.0 or later for this to work (or patch your current version).

I had the same issues running CouchDB 1.4.0 or a Raspberry Pi (raspbian jessie).

I confirmed with the following command that the CouchDB server was only sending its own certificate instead of the entire certificate chain, as mandated by the TLS spec:

openssl s_client -connect myhostname:6984 -showcerts

This showed only one certificate. In addition, it reported a verification failure. My certificate is from Namecheap. While I have the root certificate of the issuer (COMODO RSA) installed on the client machine, at least one intermediate certificate is required to complete the chain.

Note that some browsers are able to automatically fetch intermediate certs and all appears fine. Most of the command line tools (curl, perl, python, openssl) failed, though. Also interesting was that on Android's Chrome browser it occasionally showed a green lock (all fine) and at other times reported that the site could not be verified. I suspect that it's using a local certificate cache. If I happened to browse a site that was supplying the same intermediate certificates, then verification would subsequently succeed for my CouchDB server, until the cache was purged.

After digging into CouchDB and Erlang source, I found the following: Erlang can be passed a PEM file as the 'cacertfile'. This is used both for verifying client certificates (if enabled) and for composing the full certificate chain to be sent to the client in the TLS Certificate message. However, my version of CouchDB wasn't passing the cacertfile argument, unless one specified cacert_file AND verify_ssl_certificates = true. However, if those conditions are met, it omits the server key and certificate!

I found that this was already filed as bug COUCHDB-2028. Note that the bug says this has been resolved in version 1.7.0, which I believe does not exist.

I found that the fix was applied to the official CouchDB repository on 2014-01-30. Unfortunately the official revision history does not show the fix, but digging through the git repo shows that the fix was first officially released in CouchDB 1.6.0.

In my case, I was able to download and compile and install CouchDB 1.6.1 from source. Now, the above openssl command shows a chain of 4 certificates sent by the CouchDB server. I supply the server key and cert, plus the cacert_file with the CA bundle as downloaded from Namecheap. verify_ssl_certificates is false. All browsers I've tested are trusting the site, plus my commandline tools are working without hacks to disable verification.