Accessing Response Header Location in Through Python

Question

I'm currently trying to access the location field in a response header from a GET request to url: https://dbr.ee/aUJA/d?. Currently, I have been able to view the location field through this Python code:

import requests
r = requests.get('hhttps://dbr.ee/aUJA/d?', allow_redirects=False, headers={'User-Agent': 'Mozilla/5.0'})
print r.headers

But the output is the wrong location field

{'Status': '302 Found', 'X-Request-Id': '9e968067-1bee-4cc9-9305-19d45d5cb6ea', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Transfer-Encoding': 'chunked', 'Set-Cookie': '__cfduid=d21c538fd46c153a046bf461ca281978d1499637583; expires=Mon, 09-Jul-18 21:59:43 GMT; path=/; domain=.dbr.ee; HttpOnly, ahoy_visitor=f4f1c08c-add3-45c0-8325-675b1caf3048; path=/; expires=Tue, 09 Jul 2019 21:59:44 -0000, ahoy_visit=cdbb4ca8-3272-473c-8562-03596d88ec0f; path=/; expires=Mon, 10 Jul 2017 01:59:44 -0000, ahoy_track=true; path=/, SERVERID=; Expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/', 'X-Runtime': '0.006820', 'Server': 'cloudflare-nginx', 'Connection': 'keep-alive', 'Location': 'hhttps://dbr.ee/aUJA', 'Cache-Control': 'no-cache', 'Date': 'Sun, 09 Jul 2017 21:59:44 GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Type': 'text/html; charset=utf-8', 'CF-RAY': '37be8d52fdc83822-ATL'}

Which is:

'Location': 'hhttps://dbr.ee/aUJA'

While on the site, the actual response header is this (viewed through Chrome Developer Tools)

cache-control:no-cache cf-ray:37be8bacacb437d4-ATL content-type:text/html; charset=utf-8 date:Sun, 09 Jul 2017 21:58:36 GMT location:hhttps://s.dbr.ee/sffc/python%2Dlogo%2Dmaster%2Dv3%2DTM.png.zip?temp_url_sig=41ebabb749293a6fe3f3ec82c5ab8ec01b0ed053&temp_url_expires=1499637816&filename=python-logo-master-v3-TM.png.zip;&attachment server:cloudflare-nginx set-cookie:ahoy_visit=f7d15e42-155c-443f-a637-22c3681863a5; path=/; expires=Mon, 10 Jul 2017 01:58:36 -0000 set-cookie:_dbree_session=U2x6akdCbUJ4c28wdW9MeUFYOXo1QUVxLzV3ZVNxcGtTWW1jbVdkWEdPOWZPMWFiOEl4M0VWY1dOWGNYTjNubEJoVWJHejRCTlQwQlkwL0UrM09QallTMzhFZlU3RFBBTDZxaW9xcGRMeXNlQS9mZFByYTZQWTM0ZlBHMU50ekhhTkt1bjZENXJHRnc2a3dWeGY2d3BBPT0tLVNKOTJnL0Q3SjloWEc0MTZqTnRPNFE9PQ%3D%3D--2dd8f3e77a673f385c9a231af426b55f1d1f71c0; domain=dbr.ee; path=/; HttpOnly set-cookie:SERVERID=; Expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/ status:302 status:302 Found x-content-type-options:nosniff x-frame-options:SAMEORIGIN x-request-id:f57f3ca7-c7aa-4449-a2d7-7b5014010d0f x-runtime:0.015892 x-xss-protection:1; mode=block

where Location is

location:hhttps://s.dbr.ee/sffc/python%2Dlogo%2Dmaster%2Dv3%2DTM.png.zip?temp_url_sig=41ebabb749293a6fe3f3ec82c5ab8ec01b0ed053&temp_url_expires=1499637816&filename=python-logo-master-v3-TM.png.zip;&attachment

Which is the download link I am trying to scrape in Python. This appears in Developer Tools after clicking the Direct Download button.

How can I get the header to show me the correct field location in Python?

*links have been modified with h in front of http because of not allowing me to post more than 2 links, but are necessary for context of the question

I don't think requests does. I know selenium does, I tried to get the javascript from the page and run it in selenium but it didn't work, although I wasn't really sure which javascript to run to emulate clicking on the buttons. @PeterWood — Tommy McDonald

Asad Saeeduddin Asad Saeeduddin · Accepted Answer · 2017-07-09T23:45:19

Looks like the issue was the missing referer header. Once I add that to your code I get the appropriate 302 redirect response, with the correct Location header:

import requests
r = requests.get('https://dbr.ee/aUJA/d?', allow_redirects=False, headers={
    'Referer': 'https://dbr.ee/aUJA'
})
print(r.headers)

Which produces:

{'Date': 'Sun, 09 Jul 2017 23:44:55 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Set-Cookie': '__cfduid=d071cba66cc515ca7f2bc620362c6d46d1499643895; expires=Mon, 09-Jul-18 23:44:55 GMT; path=/; domain=.dbr.ee; HttpOnly, ahoy_visitor=64d9f580-781e-4037-8951-ce57b73df720; path=/; expires=Tue, 09 Jul 2019 23:44:55 -0000, ahoy_visit=802132cc-4e0e-4089-9be5-49f05223f567; path=/; expires=Mon, 10 Jul 2017 03:44:55 -0000, SERVERID=; Expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/', 'Status': '302 Found', 'Cache-Control': 'no-cache', 'X-XSS-Protection': '1; mode=block', 'X-Request-Id': '14a0d0df-c14d-477d-b87c-b6edb823619c', 'Location': 'https://s.dbr.ee/sffc/python%2Dlogo%2Dmaster%2Dv3%2DTM.png.zip?temp_url_sig=084b2b71c8c12df993d528e991a5b44e46e974ef&temp_url_expires=1499644195&filename=python-logo-master-v3-TM.png.zip;&attachment', 'X-Runtime': '0.006968', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Server': 'cloudflare-nginx', 'CF-RAY': '37bf2769fda80fa5-YYZ'}

Accessing Response Header Location in Through Python

1 Answers