Script for scraping results doesn't seem to work?

Question

This is a script I have found utilized to scrape results of graduate admissions for different programs using the grad cafe website. However, when I run it to find results on "political science" it states that I have the following error

Traceback (most recent call last):
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 57, in <module>
    data = get_data()
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 52, in get_data
    pages = get_pages()
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 45, in get_pages
    n = find_n_pages()
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 41, in find_n_pages
    reg = re.search('over\s([\d]*)\spages',html)
  File "C:\Users\lakna\AppData\Local\Programs\Python\Python36-32\lib\re.py", line 182, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object

How should I go about fixing this? Following is the code I have used

def get_page(i=0, keyword="Political Science"):
    time.sleep(10)
    if i==0:
        #To change subjects, you want to change the keyword to say biostatistics,
        #test it by searching on the site to make sure you get what you want.
        url = "http://thegradcafe.com/survey/index.php?q="+keyword+"*&t=a&o=&pp=250"
    else:
        url="http://thegradcafe.com/survey/index.php?q="+keyword+"*&t=a&pp=250&o=&p="+str(i)
    response = urlopen(url)
    html = response.read().decode('utf-8')
    return
def find_n_pages():
    html = get_page()
    reg = re.search('over\s([\d]*)\spages',html)
    return int(reg.groups()[0])

def get_pages():
    n = find_n_pages()
    print ("Getting",n,"pages.")
    pages = [get_page(i) for i in range(1,n+1)]
    return pages

def get_data():
    data=[]
    pages = get_pages()
    for page in pages:
        data=get_data_from_page(page,data)
    return data

Too much code. Please provide us a minimal reproducible example. — MooingRawr

Andy Hayden Andy Hayden · Accepted Answer · 2018-04-10T20:24:51

Your get_page function isn't returning html, it's returning None.

def get_page(i=0, keyword="Political Science"):
    ...
    html = response.read().decode('utf-8')
    return  # this is equivalent to return None (or not having this line at all)

should read:

def get_page(i=0, keyword="Political Science"):
    ...
    return response.read().decode('utf-8')

Hence the error:

In [11]: re.search("", None)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-1858f7517272> in <module>()
----> 1 re.search("", None)

/Users/andy/.miniconda3/lib/python3.6/re.py in search(pattern, string, flags)
    180     """Scan through string looking for a match to the pattern, returning
    181     a match object, or None if no match was found."""
--> 182     return _compile(pattern, flags).search(string)
    183
    184 def sub(pattern, repl, string, count=0, flags=0):

TypeError: expected string or bytes-like object

Script for scraping results doesn't seem to work?

1 Answers