replace characters not working in python

20
votes

I am using beautiful soup and I am writing a crawler and have the following code in it:

  print soup.originalEncoding
                #self.addtoindex(page, soup)


                links=soup('a')
            for link in links:

                if('href' in dict(link.attrs)):                   
                    link['href'].replace('..', '')
                    url=urljoin(page, link['href'])
                    if url.find("'") != -1:
                        continue
                    url = url.split('?')[0]
                    url = url.split('#')[0]
                    if url[0:4] == 'http':
                        newpages.add(url)
        pages = newpages

The link['href'].replace('..', '') is supposed to fix links that come out as ../contact/orderform.aspx, ../contact/requestconsult.aspx, etc. However, it is not working. Links still have the leading ".." Is there something I am missing?

3
python

3 Answers

56
votes

string.replace() returns the string with the replaced values. It doesn't modify the original so do something like this:

link['href'] = link['href'].replace("..", "")
14
votes

string.replace() returns a copy of the string with characters replaced, as strings in Python are immutable. Try

s = link['href'].replace("..", '')
url=urljoin(page, s)
9
votes

It is not an inplace replacement. You need to do:

link['href'] = link['href'].replace('..', '')

Example:

a = "abc.."
print a.replace("..","")
'abc'
 print a
'abc..'
a = a.replace("..","")
print a
'abc'