HTTPoison - getting different response than a browser

Question

I'm doing:

HTTPoison.get! "https://www.nature.com/articles/d41586-020-00741-x"

Which works fine in the browser but gives me a 303 when I try and fetch the page with HTTPoison.

What am I doing wrong?

Have you tried the :follow_redirect option (hexdocs.pm/httpoison/HTTPoison.Request.html)? — Derek Brown

sbacarob sbacarob · Accepted Answer · 2020-03-17T22:26:19

The 303 status code is a redirection code. In order to follow the redirection, you need to pass the option to HTTPoison to follow it:

HTTPoison.get!("https://www.nature.com/articles/d41586-020-00741-x", [], follow_redirect: true)

However, you most likely will need to pass a :force_redirect option to Hackney as well, since AFAIK it doesn't accept 303 as a valid redirect status code.

So, a working version would be:

HTTPoison.get!("https://www.nature.com/articles/d41586-020-00741-x",
               [],
               follow_redirect: true,
               hackney: [{:force_redirect, true}])

Which will produce:

%HTTPoison.Response{
  body: "\n\n\n\n\n\n\n\n<!DOCTYPE html>\n<html lang=\"en\" class=\"grade-c\">\n<head>\n    <meta charset=\"utf-8\">\n<link rel=\"dns-prefetch\" href=\"//ajax.googleapis.com\"/>\n<link rel=\"dns-prefetch\" href=\"//fonts.googleapis.com\"/>\n<link rel=\"dns-prefetch\" href=\"//fonts.gstatic.com\"/>\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0, shrink-to-fit=no\">\n\n    <title>What China’s coronavirus response can teach the rest of the world</title>\n    <meta name=\"description\" content=\"Researchers are studying the effects of China&#x27;s lockdowns to glean insights about controlling the viral pandemic.\"/>\n    <meta property=\"og:url\" content=\"https://www.nature.com/articles/d41586-020-00741-x\"/>\n    <meta property=\"og:type\" content=\"article\"/>\n    <meta property=\"og:title\" content=\"What China’s coronavirus response can teach the rest of the world\"/>\n    <meta property=\"og:description\" content=\"Researchers are studying the effects of China&#x27;s lockdowns to glean insights about controlling the viral pandemic.\"/>\n    <meta property=\"og:image\"\n          content=\"https://media.nature.com/lw1024/magazine-assets/d41586-020-00741-x/d41586-020-00741-x_17788864.jpg\"/>\n    <meta name=\"twitter:card\" content=\"summary_large_image\"/>\n    <meta name=\"twitter:site\" content=\"@nature\"/>\n    <meta name=\"twitter:title\" content=\"What China’s coronavirus response can teach the rest of the world\"/>\n    <meta name=\"twitter:description\" content=\"Researchers are studying the effects of China&#x27;s lockdowns to glean insights about controlling the viral pandemic.\"/>\n    <meta name=\"twitter:image\"\n          content=\"https://media.nature.com/lw1024/magazine-assets/d41586-020-00741-x/d41586-020-00741-x_17788864.jpg\"/>\n    \n\n    <meta name=\"journal_id\" content=\"41586\"/>\n\n    <meta name=\"dc.title\" content=\"What China&#8217;s coronavirus response can teach the rest of the world\"/>\n\n    <meta name=\"dc.source\" content=\"Nature 2020\"/>\n\n    <meta name=\"dc.format\" content=\"text/html\"/>\n\n    <meta name=\"dc.publisher\" content=\"Nature Publishing Group\"/>\n\n    <meta name=\"dc.date\" content=\"2020-03-17\"/>\n\n    <meta name=\"dc.type\" content=\"News Explainer\"/>\n\n    <meta name=\"dc.language\" content=\"En\"/>\n\n    <meta name=\"dc.copyright\" content=\"2020 Nature\"/>\n\n    <meta name=\"dc.rightsAgent\" content=\"[email protected]\"/>\n\n    <meta name=\"dc.description\" content=\"Researchers are studying the effects of China&#39;s lockdowns to glean insights about controlling the viral pandemic.  Researchers are studying the effects of China&#39;s lockdowns to glean insights about controlling the viral pandemic.\"/>\n\n    <meta name=\"prism.publicationName\" content=\"Nature\"/>\n\n    <meta name=\"prism.publicationDate\" content=\"2020-03-17\"/>\n\n    <meta name=\"prism.section\" content=\"News\"/>\n\n    <meta name=\"prism.startingPage\" content=\"\"/>\n\n    <meta name=\"prism.endingPage\" content=\"\"/>\n\n    <meta name=\"prism.copyright\" content=\"2020 Nature\"/>\n\n    <meta name=\"prism.rightsAgent\" content=\"[email protected]\"/>\n\n    <meta name=\"prism.url\" content=\"https://www.nature.com/articles/d41586-020-00741-x\"/>\n\n    <meta name=\"prism.doi\" content=\"doi:10.1038/d41586-020-00741-x\"/>\n\n    <meta name=\"dc.identifier\" content=\"doi:10.1038/d41586-020-00741-x\"/>\n\n    <meta name=\"DOI\" content=\"10.1038/d41586-020-00741-x\"/>\n\n    <meta name=\"description\" content=\"Researchers are studying the effects of China&#39;s lockdowns to glean insights about controlling the viral pandemic.  Researchers are studying the effects of China&#39;s lockdowns to glean insights about controlling the viral pandemic.\"/>\n\n    <meta name=\"dc.creator\" content=\"David Cyranoski\"/>\n\n    <meta name=\"dc.subject\" content=\"Government\"/>\n\n    <meta name=\"dc.subject\" content=\"Infection\"/>\n\n    <meta name=\"dc.subject\" content=\"Public health\"/>\n\n\n\n<script>(function(e){var t=e.documentElement,n=e.implementation;t.className='js';if(n&&n.hasFeature('http://www.w3.org/TR/SVG11/feature#Image','1.1')){t.className+=' svg'}})(document)</script>\n<link rel=\"stylesheet\" href=\"/static/css/m" <> ...,
  headers: [
    ...
  ],
  request: %HTTPoison.Request{
    body: "",
    headers: [],
    method: :get,
    options: [follow_redirect: true, hackney: [force_redirect: true]],
    params: %{},
    url: "https://www.nature.com/articles/d41586-020-00741-x"
  },
  request_url: "https://www.nature.com/articles/d41586-020-00741-x",
  status_code: 200
}

HTTPoison - getting different response than a browser

1 Answers