python, "urlparse.urlparse(url).hostname" return None value

6
votes

After loging in on a website I want to collect its links. This I do with this function (using mechanize and urlparse libraries):

br = mechanize.Browser()

.
. #logging in on website
.

for link in br.links():
    url = urlparse.urljoin(link.base_url, link.url)

    hostname = urlparse.urlparse(url).hostname
    path = urlparse.urlparse(url).path

    #print hostname #by printing this I found it to be the source of the None value

    mylinks.append("http://" + hostname + path)

and I get this error message:

    mylinks.append("http://" + hostname + path)
TypeError: cannot concatenate 'str' and 'NoneType' objects

I am not sure on how to fix this, or even if it can be fixed at all. Is there any way to force the function to append even if it would produce a nonworking and weird result for the None value?

Alternatively, what I'm really after in the link is what the link ends with. for example, the html code for one of the links look like this (what I am after is the world "lexik"):

<td class="center">
    <a href="http://UnimportantPartOfLink/lexik>>lexik</a>
</td>

so an alternative route would be if mechanize can just collect this value directly, bypassing the links and None value troubles

2
pythonurlparse
Can't you just wrap the append statement in a try except block? – RobinL

2 Answers

6
votes

Another good way without any try and except block -

Replace hostname = urlparse.urlparse(url).hostname with 

hostname = urlparse.urlparse(url).hostname or ''

and similarly path = urlparse.urlparse(url).path with 

path = urlparse.urlparse(url).path or ''

Hope this helps !

4
votes

Why not use a try/except block?

try:
    mylinks.append("http://" + hostname + path)
except TypeError:
    continue

If there's an error, it would just skip the appending and go on with the loop.

Hope this helps!