IndexError: list index out of range in a function using BeautifulSoup

Question

I am making a GUI based project in Python with the module tkinter. It fetches basic data from online judge, eg, SPOJ using Beautiful Soup. I am a newbie in Python, so most of the things I have written is with basic tutorials from internet. However, for a particular piece of code, I am totally stuck up.

import sys
import urllib.request
from bs4 import BeautifulSoup 
import re

userName = 'xilinx'
spojUrl = 'http://www.spoj.com/users/'+userName

with urllib.request.urlopen(spojUrl) as x:
   html = x.read()
soup = BeautifulSoup(html, 'html.parser')
# li - list of successful submissions
li = soup.find_all('table', class_='table table-condensed')[0].find_all('td')
listOfSolvedProblemCodes = []
    for submission in li:
        problemCode = submission.get_text()
        if problemCode:
            listOfSolvedProblemCodes.append(problemCode)
print (userName+ ' has solved',len(listOfSolvedProblemCodes),'problems on Spoj.')

This part of code works fine when I run it with python submissions.py

After testing this part, I try to incorporate it into the larger code, where the problem arises. I am including here the relevant section of the code:

In frame.py:

def compStats ():
    if ch == "SPOJ":
        stats.show(ch, userName)

B2 = tkinter.Button(root, text="My Statistics", command=compStats)
B2.place(anchor = W, x = 30, y = 220, width=200)

In stats.py:

def show(ch, userName):

    if ch == 'SPOJ':

        spojUrl = 'http://www.spoj.com/users/'+userName

        with urllib.request.urlopen(spojUrl) as x:
            html = x.read()
        soup = BeautifulSoup(html, 'html.parser')
        li = soup.find_all('table', class_='table table-condensed')[0].find_all('td')
        listOfSolvedProblemCodes = []
        for submission in li:
            problemCode = submission.get_text()
            if problemCode:
                listOfSolvedProblemCodes.append(problemCode)


    # then collect more information from soup and print it through labels in another window

    det = tkinter.Tk()
    det.title("Statistics")
    det.geometry("800x600")

But the problem of IndexError occurs in stats.py in the line:

li = soup.find_all('table', class_='table table-condensed')[0].find_all('td')

Exception in Tkinter callback

Traceback (most recent call last):

File "C:\Users\Aa\AppData\Local\Programs\Python\Python35-32\lib\tkinter__init .py", line 1550, in __call

return self.func(*args)

File "frame.py", line 34, in compStats

stats.show(ch, userName)

File "C:\Users\Aa\AppData\Local\Programs\Python\Python35-32\stats.py", line 17, in show

li = soup.find_all('table', class_='table table-condensed')[0].find_all('td')

IndexError: list index out of range

I am unable to understand why the code is unable to work here. Please help!

Bryan Oakley Bryan Oakley · Accepted Answer · 2016-07-01T12:58:37

The first step in debugging this is to take the complex line that is throwing an error and make it simpler. You can then inspect the intermediate values so see if the assumptions you're making about the code are true. In this case, your assumption is that soup.find_all('table', ...) is actually finding something.

For example, change this:

li = soup.find_all('table', class_='table table-condensed')[0].find_all('td')

to this:

tables = soup.find_all('table', class_='table table-condensed')
li = tables[0].find_all('td')

Next, add a print statement to examine tmp:

print("tables is", tables)

You will find that tables is likely empty, so when you try to do tables[0] you get an error since index 0 is out of range.

IndexError: list index out of range in a function using BeautifulSoup

1 Answers