Web Scraping with Python: RuntimeError maximum recursion depth exceeded

Question

I'm trying to scape some data from a website that updates frequently (here I'm using AAPL stock off of Yahoo finance). This code doesn't run. I've tested every part individually, but I still get an error saying "RuntimeError: maximum recursion depth exceeded".
Thanks in advance for your help troubleshooting!

import time  
import lxml, requests  
from bs4 import BeautifulSoup  

url= "http://finance.yahoo.com/q?uhb=uh3_finance_vert&fr=&type=2button&s=aapl"  

def PrintPriceAAPL():  
    r = requests.get(url)  
    soup = BeautifulSoup(r.content, "lxml")  
    print (soup.find(id="yfs_l84_aapl").string) #ID of current stock price  
    time.sleep(60)  

while True:  
    PrintPriceAAPL()

EDIT: This is the full error:

Traceback (most recent call last):
  File "C:/Users/sjung/Documents/PrintPriceAAPL.py", line 15, in <module>
    PrintPriceAAPL()
  File "C:/Users/sjung/Documents/PrintPriceAAPL.py", line 11, in PrintPriceAAPL
    print (soup.find_all(id="yfs_l84_aapl")[0].string) #ID of current stock price
  File "C:\Python27\lib\idlelib\rpc.py", line 595, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "C:\Python27\lib\idlelib\rpc.py", line 210, in remotecall
    seq = self.asynccall(oid, methodname, args, kwargs)
  File "C:\Python27\lib\idlelib\rpc.py", line 225, in asynccall
    self.putmessage((seq, request))
  File "C:\Python27\lib\idlelib\rpc.py", line 324, in putmessage
    s = pickle.dumps(message)
  File "C:\Python27\lib\copy_reg.py", line 74, in _reduce_ex
    getstate = self.__getstate__
RuntimeError: maximum recursion depth exceeded

Just from what you posted, your loop never ends. Is there more code to show? Because that is most likely your problem if that is all your code. — idjaw
Once the URL is corrected to http://finance.yahoo.com/q?uhb=uh3_finance_vert&fr=&type=2button&s=aapl your code works fine for me. Update your question to include the full traceback so we can see where the problem is. You could try using the default html parser with soup = BeautifulSoup(r.content, "html.parser") to see whether the error is something to do with the lxml parser. — mhawke
I changed the link, but even then I get the same error. I also get this error with "html.parser" — Max Smith

sgrg sgrg · Accepted Answer · 2015-09-29T22:34:28

find() searches for children recursively. So, find() will look at all of soup's descendants: its children, childrens' children, etc.

Each time the next level of children is examined, the "recursion depth" increases for the run-time stack. If you are only interested in immediate children, find() has a recursive=False argument which should only examine one level down.

Alternatively, you could increase the recursion depth but be careful. Reference this post: What is the maximum recursion depth in Python, and how to increase it?

For more information on what the run-time stack is, look at: https://chortle.ccsu.edu/AssemblyTutorial/Chapter-25/ass25_9.html

Web Scraping with Python: RuntimeError maximum recursion depth exceeded

1 Answers