PhantomJS is returning status fail when page being loaded has an invalid content-type response header

Question

I encountered a very weird error today within phantomjs, that I'm turning to the community for. The expected JSON response of an API I was invoking with phantomJS was not being returned, but rather the result was a status of failure.

onResourceReceived() showed a 200 http status code for the resource in question onLoadFinished() shows a status of fail

After debugging this for sometime, I noticed that the site was returning a non standard content-type header on the response. Rather than content-type of "application/json" the header being returned was "application/servicename-1.0+json".

To verify this, we spun up a local webserver that served a similar header, and sure enough phantom js cannot load the page. Setting the response header to "application/json" and phantomjs correctly renders the page and sets the page objects page.plainText variable. I've included the testing script below.

Has anyone encountered anything like this before?

Any suggestions on how to handle this issue within phantomjs?

Vaviloff Vaviloff · Accepted Answer · 2016-07-19T06:46:31

The simplest and quickest solution I can think of (not involving editing PhantomJS' source code and compiling it) is to set up a simple local proxy server in front of PhantomJS that would rewrite the incorrect header.

It could be something like fiddler2, Charles Proxy or a simple node.js script, like this quick-n-dirty example:

// npm install proxy-tamper
var proxy = require('proxy-tamper').start({port: 3000});

proxy.tamper(/api.truelocal.com.au.*$/, function (request) {

  request.onResponse(function (response) {

    if ('content-type' in response.headers && response.headers['content-type'] == 'application/servicename-1.0+json') {
        response.headers['content-type'] = 'application/json';
    }

    response.complete();
  }); 
});

Then have PhantomJS use it:

 phantomjs --proxy=127.0.0.1:3000 script.js

Note that this won't work for secure pages.

PhantomJS is returning status fail when page being loaded has an invalid content-type response header

2 Answers