python requests link headers

26
votes

I'm trying to find best way to capture links listed under response headers, exactly like this one and I'm using python requests module. Below is link which has Link Headers section on Python Requests page: docs.python-requests.org/en/latest/user/advanced/

But, in my case my response headers contains links like below:

{'content-length': '12276', 'via': '1.1 varnish-v4', 'links': '<http://justblahblahblah.com/link8.html>;rel="last">,<http://justblahblahblah.com/link2.html>;rel="next">', 'vary': 'Accept-Encoding, Origin'}

Please notice > after "last" which is not the case under Requests examples and I just cant seem to figure out how to solve this.

2
pythonhyperlinkheaderpython-requests
What is your question? - Alik
Apologize if I wasn't clear, but I want to know how I can get Next and last link from response using rel in header response. - user1819085
If requests do not work with this link header then you can parse it manually: it is not that hard. - Alik
Use request's parsing function as a reference. See github.com/kennethreitz/requests/blob/… - Alik
actually Im already doing that now, using re and looking for pattern. First, I'm parsing links exact links and then looking for page value(above links are just examples). But I guess I should rather look for pattern with link and rel - user1819085

2 Answers

69
votes

There is already a way provided by requests to access links header

response.links

It returns the dictionary of links header value which can easily parsed further using 

response.links['next']['url']

to get the required values.

13
votes

You can parse the header's value manually. To make things easier you might want to use request's parsing function parse_header_links as a reference.

Or you can do some find/replace and use original parse_header_links

In [1]: import requests

In [2]: d = {'content-length': '12276', 'via': '1.1 varnish-v4', 'links': '<http://justblahblahblah.com/link8.html>;rel="last">,<http://justblahblahblah.com/link2.html>;rel="next">', 'vary': 'Accept-Encoding, Origin'}

In [3]: requests.utils.parse_header_links(d['links'].rstrip('>').replace('>,<', ',<'))
Out[3]:
[{'rel': 'last', 'url': 'http://justblahblahblah.com/link8.html'},
 {'rel': 'next', 'url': 'http://justblahblahblah.com/link2.html'}]

If there might be a space or two between >, and < then you need to do replace with a regular expression.