15
votes

Why does request not download a response for this webpage?

#!/usr/bin/python

import requests

headers={ 'content-type':'application/x-www-form-urlencoded; charset=UTF-8',
     'Accept-Encoding': 'gzip, deflate',
     'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0',
     'Referer' : 'http://sportsbeta.ladbrokes.com/football',
    }

payload={'N': '4294966750',
     'facetCount_156%23327': '12',
     'facetCount_157%23325': '8',
     'form-trigger':'moreId',
     'moreId':'156%23327',
     'pageId':'p_football_home_page',
     'pageType':'EventClass',
     'type':'ajaxrequest'
     }

url='http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController'

r = requests.post(url, data=payload, headers=headers)

These are the parameters of the POST that I see in Firebug, and there the response received back contains a list (of football leagues), yet when I run my python script like this I get nothing.

(you can see the request in Firefox by clicking the See All in the competitions section of the left hand nav bar of link and looking at the XHR in Firebug. The Firebug response shows the HTML body as expected.)

Anyone any ideas? Will my handling of the % symbols in the payload be causing any trouble at all?

EDIT: Attempt using session

from requests import Request, Session

#turn post string into dict: 
def parsePOSTstring(POSTstr):
    paramList = POSTstr.split('&')
    paramDict = dict([param.split('=') for param in paramList])
    return paramDict

headers={'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0',
     'Referer' : 'http://sportsbeta.ladbrokes.com/football'
    }

#prep the data (POSTstr copied from Firebug raw source)
POSTstr = "moreId=156%23327&facetCount_156%23327=12&event=&N=4294966750&pageType=EventClass&
          pageId=p_football_home_page&type=ajaxrequest&eventIDNav=&removedSelectionNav=&
          currentSelectedId=&form-trigger=moreId"
payload = parsePOSTstring(POSTstr)

#end url
url='http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController'

#start a session to manage cookies, and visit football page first so referer agrees
s = Session()
s.get('http://sportsbeta.ladbrokes.com/football')
#now visit disired url with headers/data
r = s.post(url, data=payload, headers=headers)

#print output
print r.text #this is empty

Working curl

curl 'http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController'
-H 'Cookie: JSESSIONID=DE93158F07E02DD3CC1CC32B1AA24A9E.ecomprodsw015;
    geoCode=FRA; 
    FLAGS=en|en|uk|default|ODDS|0|GBP;
    ECOM_BETA_SPORTS=1;
    PLAYED=4%7C0%7C0%7C0%7C0%7C0%7C0'
-H 'Referer: http://sportsbeta.ladbrokes.com/football'
-H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) 
    Gecko/20100101 Firefox/27.0'  
--data 'facetCount_157%23325=8&moreId=156%23327&
        facetCount_156%23327=12&event=&
        N=4294966750&
        pageType=EventClass&pageId=p_football_home_page&
        type=ajaxrequest&eventIDNav=&
        removedSelectionNav=&currentSelectedId=&
        form-trigger=moreId' --compressed

Yet this curl works.

1
You first need to visit http://sportsbeta.ladbrokes.com/football (instead of the homepage). Then it seems to work. You don't need any other headers aside from Referer and User-Agent. - Blender
@Blender I updated my answer with the minimal header you suggested, and also using a requests session to manage cookies and visit the football homepage first because making the ajax request, but I'm still getting an empty r.text, does this code run for you? - fpghost
It works if you properly decode the percent-encoded characters (i.e. change %23 to # before decoding it, or fix parsePOSTstring). I wasn't seeing the problem because I used a dictionary the whole time. - Blender

1 Answers

24
votes

Here's the smallest working example that I can come up with:

from requests import Session

session = Session()

# HEAD requests ask for *just* the headers, which is all you need to grab the
# session cookie
session.head('http://sportsbeta.ladbrokes.com/football')

response = session.post(
    url='http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController',
    data={
        'N': '4294966750',
        'form-trigger': 'moreId',
        'moreId': '156#327',
        'pageType': 'EventClass'
    },
    headers={
        'Referer': 'http://sportsbeta.ladbrokes.com/football'
    }
)

print response.text

You just weren't decoding the percent-encoded POST data properly, so # was being represented as %23 in the actual POST data (e.g. 156%23327 should've been 156#327).