How can I get LWP to validate SSL server certificates?

Question

How can I get LWP to verify that the certificate of the server I'm connecting to is signed by a trusted authority and issued to the correct host? As far as I can tell, it doesn't even check that the certificate claims to be for the hostname I'm connecting to. That seems like a major security hole (especially with the recent DNS vulnerabilities).

Update: It turns out what I really wanted was HTTPS_CA_DIR, because I don't have a ca-bundle.crt. But HTTPS_CA_DIR=/usr/share/ca-certificates/ did the trick. I'm marking the answer as accepted anyway, because it was close enough.

Update 2: It turns out that HTTPS_CA_DIR and HTTPS_CA_FILE only apply if you're using Net::SSL as the underlying SSL library. But LWP also works with IO::Socket::SSL, which will ignore those environment variables and happily talk to any server, no matter what certificate it presents. Is there a more general solution?

Update 3: Unfortunately, the solution still isn't complete. Neither Net::SSL nor IO::Socket::SSL is checking the host name against the certificate. This means that someone can get a legitimate certificate for some domain, and then impersonate any other domain without LWP complaining.

Update 4: LWP 6.00 finally solves the problem. See my answer for details.

Answer 1

This long-standing security hole has finally been fixed in version 6.00 of libwww-perl. Starting with that version, by default LWP::UserAgent verifies that HTTPS servers present a valid certificate matching the expected hostname (unless $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} is set to a false value or, for backwards compatibility if that variable is not set at all, either $ENV{HTTPS_CA_FILE} or $ENV{HTTPS_CA_DIR} is set).

This can be controlled by the new ssl_opts option of LWP::UserAgent. See that link for details on how the Certificate Authority certificates are located. But be careful, the way LWP::UserAgent used to work, if you provide a ssl_opts hash to the constructor, then verify_hostname defaulted to 0 instead of 1. (This bug was fixed in LWP 6.03.) To be safe, always specify verify_hostname => 1 in your ssl_opts.

So use LWP::UserAgent 6; should be sufficient to have server certificates validated.

Answer 2

There are two means of doing this depending on which SSL module you have installed. The LWP docs recommend installing Crypt::SSLeay. If that's what you've done, setting the HTTPS_CA_FILE environment variable to point to your ca-bundle.crt should do the trick. (the Crypt::SSLeay docs mentions this but is a bit light on details). Also, depending on your setup, you may need to set the HTTPS_CA_DIR environment variable instead.

Example for Crypt::SSLeay:

use LWP::Simple qw(get);
$ENV{HTTPS_CA_FILE} = "/path/to/your/ca/file/ca-bundle";
$ENV{HTTPS_DEBUG} = 1;

print get("https://some-server-with-bad-certificate.com");

__END__
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL3 alert write:fatal:unknown CA
SSL_connect:error in SSLv3 read server certificate B
SSL_connect:error in SSLv3 read server certificate B
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL3 alert write:fatal:bad certificate
SSL_connect:error in SSLv3 read server certificate B
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:error in SSLv2 read server hello B

Note that get doesn't die, but it does return an undef.

Alternatively, you can use the IO::Socket::SSL module (also available from the CPAN). To make this verify the server certificate you need to modify the SSL context defaults:

use IO::Socket::SSL qw(debug3);
use Net::SSLeay;
BEGIN {
    IO::Socket::SSL::set_ctx_defaults(
        verify_mode => Net::SSLeay->VERIFY_PEER(),
        ca_file => "/path/to/ca-bundle.crt",
      # ca_path => "/alternate/path/to/cert/authority/directory"
    );
}
use LWP::Simple qw(get);

warn get("https:://some-server-with-bad-certificate.com");

This version also causes get() to return undef but prints a warning to STDERR when you execute it (as well as a bunch of debugging if you import the debug* symbols from IO::Socket::SSL):

% perl ssl_test.pl
DEBUG: .../IO/Socket/SSL.pm:1387: new ctx 139403496
DEBUG: .../IO/Socket/SSL.pm:269: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:271: socket connected
DEBUG: .../IO/Socket/SSL.pm:284: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:327: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1135: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

DEBUG: .../IO/Socket/SSL.pm:333: fatal SSL error: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
DEBUG: .../IO/Socket/SSL.pm:1422: free ctx 139403496 open=139403496
DEBUG: .../IO/Socket/SSL.pm:1425: OK free ctx 139403496
DEBUG: .../IO/Socket/SSL.pm:1135: IO::Socket::INET configuration failederror:00000000:lib(0):func(0):reason(0)
500 Can't connect to some-server-with-bad-certificate.com:443 (SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed) 

Answer 3

I landed on this page looking for a way to bypass SSL validation but all answers were still very helpful. Here are my findings. For those looking to bypass SSL validation (not recommended but there may be cases where you will absolutely have to), I'm on lwp 6.05 and this worked for me:

use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common qw(GET);
use Net::SSL;

my $ua = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0 }, );
my $req = GET 'https://github.com';
my $res = $ua->request($req);
if ($res->is_success) {
    print $res->content;
} else {
    print $res->status_line . "\n";
}

I also tested on a page with POST and it also worked. The key is to use Net::SSL along with verify_hostname = 0.

Answer 4

If you use LWP::UserAgent directly (not via LWP::Simple) you can validate the hostname in the certificate by adding the "If-SSL-Cert-Subject" header to your HTTP::Request object. The value of the header is treated as a regular expression to be applied on the certificate subject, and if it does not match, the request fails. For example:

#!/usr/bin/perl 
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new(GET => 'https://yourdomain.tld/whatever');
$req->header('If-SSL-Cert-Subject' => '/CN=make-it-fail.tld');

my $res = $ua->request( $req );

print "Status: " . $res->status_line . "\n"

will print

Status: 500 Bad SSL certificate subject: '/C=CA/ST=Ontario/L=Ottawa/O=Your Org/CN=yourdomain.tld' !~ //CN=make-it-fail.tld/

Answer 5

All the solutions presented here contain a major security flaw in that they only verify the validity of the certificate's trust chain, but don't compare the certificate's Common Name to the hostname you're connecting to. Thus, a man in the middle may present an arbitrary certificate to you and LWP will happily accept it as long as it's signed by a CA you trust. The bogus certificate's Common Name is irrelevant because it's never checked by LWP.

If you're using IO::Socket::SSL as LWP's backend, you can enable verification of the Common Name by setting the verifycn_scheme parameter like this:

use IO::Socket::SSL;
use Net::SSLeay;
BEGIN {
    IO::Socket::SSL::set_ctx_defaults(
        verify_mode => Net::SSLeay->VERIFY_PEER(),
        verifycn_scheme => 'http',
        ca_path => "/etc/ssl/certs"
    );
}

Answer 6

You may also consider Net::SSLGlue ( http://search.cpan.org/dist/Net-SSLGlue/lib/Net/SSLGlue.pm ) But, take care, it depends on recent IO::Socket::SSL and Net::SSLeay versions.

Answer 7

You are right to be concerned about this. Unfortunately, I don't think it's possible to do it 100% securely under any of the low-level SSL/TLS bindings I looked at for Perl.

Essentially you need to pass in the hostname of the server you want to connect to the SSL library before the handshaking gets underway. Alternatively, you could arrange for a callback to occur at the right moment and abort the handshake from inside the callback if it doesn't check out. People writing Perl bindings to OpenSSL seemed to have troubles making the callback interface consistently.

The method to check the hostname against the server's cert is dependent on the protocol, too. So that would have to be a parameter to any perfect function.

You might want to see if there are any bindings to the Netscape/Mozilla NSS library. It seemed pretty good at doing this when I looked at it.

Answer 8

Just perform execute the following command in Terminal: sudo cpan install Mozilla::CA

It should solve it.