How to solve webscraping code that is downloading too slow

0
votes

I'm trying to webscrape some information with python from an online database and put it into one data file in the end. at the moment I'm iterating through a database. At first I tested it for some specific websites it worked just fine, when I ran each line separately, but when I try to run the whole code it doesn't work anymore. I've now realized that the problem is that the code doesn't download the information fast enough, i.e. in this code excerpt, between line 2 and 3 there is the problem which causes it to got to except.

try:
    download = driver.find_element(By.XPATH,'...').click()
    oldfile = old_path + 'file.xlsx'
    newfile = 'output/FederalExperience_'+z1+'.xlsx'
    shutil.move(oldfile,newfile)
except:
    print("No such section")

Is there a way to tell the code to only run, when the file is downloaded? or some other way to make it work? I'm using a mac and vscode btw.

pythonweb-scrapingselenium-chromedriver
Never use bare except: like that. You don't know what the error that's happening is – chances are it's not about slowness at all. For instance, it could be a MemoryError if you're running out of memory, or an OSError if the move() fails. (It could even be you pressing ctrl+c – that'd raise a KeyboardInterrupt!) - AKX
replace the except with except Exception as e and then in the next line: print(type(e).__name__, e) You will get info what's actual error. - Mikaelblomkvistsson
In other words, get rid of that entire try:except: around your code and tell us what the actual error is you're getting. - AKX
It would be really hard for anyone to help you if you do not post the url you are trying to get the data from. - platipus_on_fire
so I tried Mikael's suggestion and I get the following error, the thing is again, when I run the code line by line separately it works, when I wait until the file is downloaded. Else I get this error message: FileNotFoundError [Errno 2] No such file or directory: '/.../Downloads/Parliamentarian Profile - Federal Experience without Parliament.xlsx' - futur3boy