Keeping a session in python while making HTTP requests

Question

I need to write a python script that makes multiple HTTP requests to the same site. Unless I'm wrong (and I may very well be) urllib reauthenticates for every request. For reasons I won't go into I need to be able to authenticate once and then use that session for the rest of my requests.

I'm using python 2.3.4

Answer 1

If you want to keep the authentication you need to reuse the cookie. I'm not sure if urllib2 is available in python 2.3.4 but here is an example on how to do it:

req1 = urllib2.Request(url1)
response = urllib2.urlopen(req1)
cookie = response.headers.get('Set-Cookie')

# Use the cookie is subsequent requests
req2 = urllib2.Request(url2)
req2.add_header('cookie', cookie)
response = urllib2.urlopen(req2)

Answer 2

Use Requests library. From http://docs.python-requests.org/en/latest/user/advanced/#session-objects :

The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance.

s = requests.session()

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")

print r.text
# '{"cookies": {"sessioncookie": "123456789"}}'

Answer 3

Python 2

If this is cookie based authentication use HTTPCookieProcessor:

import cookielib, urllib2
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")

If this is HTTP authentication use basic or digest AuthHandler:

import urllib2
# Create an OpenerDirector with support for Basic HTTP Authentication...
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(realm='PDQ Application',
                          uri='https://mahler:8092/site-updates.py',
                          user='klem',
                          passwd='kadidd!ehopper')
opener = urllib2.build_opener(auth_handler)
# ...and install it globally so it can be used with urlopen.
urllib2.install_opener(opener)
urllib2.urlopen('http://www.example.com/login.html')

... and use same opener for every request.

Python 3

In Python3 urllib2 and cookielib were moved to http.request and http.cookiejar respectively.