Enable SSL connection for https in cURL PHP (header blank)

Question

I am writing a cURL script to access the current days interest rate from the Fannie Mae website which is https. I havent been able to get past the CURLOPT_SSL_VERIFYPEER, true); option.

No username or password is required, however I need SSL verification turned on.

Testing on XAMPP dev server.

I have downloaded the .crt and .pem certs from the website using FF and saved them in the same source dir and pointed to both using CURLOPT_CAINFO, no luck

I downloaded the latest cacert.pem file from http://curl.haxx.se/ca/cacert.pem and pointed to that as well using CURLOPT_CAINFO, no luck.

If I turn CURLOPT_SSL_VERIFYPEER, to false I can retrieve the header (see below), however when I set it to true there is no header.

Tried about 7-8 solutions found by searching on here along with reading the php documention on cURL and trying several workarounds listed there, no luck.

I need to be able to retrieve the header and eventually the body using CURLOPT_SSL_VERIFYPEER, true

Any help is appreciated.

<?php

// script is designed to access an https site and retrieve the last table showing the most recent 90 day commitment for the Fannie Mae 30 year fixed rate mortgage.  Site is designed to work with cookies and has a valid SSL cert.

//turn error reporting on
error_reporting(E_ALL); ini_set("display_errors", 1); 

// cookie file name/location
$cookie_file_path = "cookies.txt";

// verify if cookie file is accessible and writable
if (! file_exists($cookie_file_path) || ! is_writable($cookie_file_path))
{
    echo 'Cookie file missing or not writable.';
    exit;
}

// url connection
$url = "https://www.fanniemae.com/content/datagrid/hist_net_yields/cur30.html";

// Initiate connection
$ch = curl_init();

// Set cURL and other options
curl_setopt($ch, CURLOPT_URL, $url); // set url
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"); // set browser/user agent
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // automatically follow Location: headers (ie redirects)
curl_setopt($ch, CURLOPT_AUTOREFERER, 1); // auto set the referer in the event of a redirect
curl_setopt($ch, CURLOPT_MAXREDIRS, 5); // make sure we dont get stuck in a loop
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10); // 10s timeout time for cURL connection
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // allow https verification if true
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2); // check common name and verify with host name
curl_setopt($ch, CURLOPT_SSLVERSION,3); // verify ssl version 2 or 3
curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "VeriSignClass3PublicPrimaryCertificationAuthority-G5.pem"); // allow ssl cert direct comparison
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'read_header'); // get header
curl_setopt($ch, CURLOPT_NOBODY, true); // exclude body
curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE); // set new cookie session
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path); // file to save cookies in
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path); // file to read cookies in

// grab URL and pass it to the browser
curl_exec($ch);

// close cURL connection, save cookie file, free up system resources
curl_close($ch);

// show header
function read_header($ch, $string) {
    print "Received header: $string";
    return strlen($string);
}
?>

This is the header that is received if CURLOPT_SSL_VERIFYPEER is set to false, blank if true

Received header: HTTP/1.1 200 OK Received header: Date: Thu, 19 Sep 2013 00:40:16 GMT Received header: Server: Apache Received header: Set-Cookie: JSESSIONID=4297C1E1760A836F691FE821FBF8B805.cportal-cl01; Path=/; Secure; HttpOnly Received header: Cache-Control: no-store Received header: Expires: Wed, 31 Dec 1969 23:59:59 GMT Received header: Pragma: no-cache Received header: X-FRAME-OPTIONS: SAMEORIGIN Received header: Content-Language: en-US Received header: Content-Length: 9344 Received header: Content-Type: text/html;charset=ISO-8859-1 Received header:

I just discovered that the OSX version of curl in mavericks (and i assume above) breaks SSL calls when there is no intermediates (ie your pem file). See: curl.haxx.se/mail/archive-2013-10/0036.html. I'm not sure if XAMPP on OSX uses it's own curl or OSX's. MAMP uses its own (which meant i had to run my script using one of the MAMP PHP interpreters) — stephendwolff

subroutines subroutines · Accepted Answer · 2013-09-19T03:02:01

You're excluding the body by using curl_setopt($ch, CURLOPT_NOBODY, true);. And I don't think you need to install certificate on your machine. The following few lines will give you everything.

$url = 'https://www.fanniemae.com/content/datagrid/hist_net_yields/cur30.html';
$ch = curl_init();    
curl_setopt($ch, CURLOPT_URL, $url); // set url
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);  
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"); // set browser/user agent    
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'read_header'); // get header
curl_exec($ch);

function read_header($ch, $string) {
    print "Received header: $string";
    return strlen($string);
}

Enable SSL connection for https in cURL PHP (header blank)

1 Answers