Cleaning HTML using python

Question

I have the below code however i am receiving an error. I am trying to get the text from an html file between Tag1 and Tag2 without the for loop the code is working (for one file) however when looping in a directory it is not

from bs4 import BeautifulSoup
from urllib import urlopen
import os
import bleach
import re
rootdir = mydirectory
for subdir, dirs, files in os.walk(rootdir):
    for file in files:
        url = file
        print url
        raw = urlopen(url).read()
        type(raw)
        Tag1 = raw.find("""<div class="song-text">""")
        Tag2 = raw.rfind("""<div style="text-align:center;padding-bottom:10px;">""")
        Cleaned = raw[Tag1+23:Tag2]
        print Cleaned

Error message: Traceback (most recent call last): File "TestClean.py", line 12, in raw = urlopen(url).read() File "/usr/lib/python2.7/urllib.py", line 87, in urlopen return opener.open(url) File "/usr/lib/python2.7/urllib.py", line 208, in open return getattr(self, name)(url) File "/usr/lib/python2.7/urllib.py", line 463, in open_file return self.open_local_file(url) File "/usr/lib/python2.7/urllib.py", line 477, in open_local_file raise IOError(e.errno, e.strerror, e.filename) IOError: [Errno 2] No such file or directory: 'paroles-a-beautiful-lie.html'

I think you try to open the file at the wrong location. url is just the filename. You might want to use os.path.join to concatenate subdir and url. BTW, use a HTML parser to parse HTML. — Matthias

GennadyS GennadyS · Accepted Answer · 2014-05-23T07:21:14

Error message indicates a lack of file. os.walk returns only the name of the file, but not the full path to it. 1) Take the path = os.path.join(subdir, file) 2) Read the file open(path).read()without urlopen

Cleaning HTML using python

2 Answers