Screen Scraping a Javascript based webpage in Python

4
votes

I am working on a screen scraping tool in Python. But, as I look through the source of the webpage, I noticed that most of the data is coming through Javascript.

Any idea, how to scrape javascript based webpage ? Any tool in Python ?

Thanks

3
pythonscreen-scrapingbeautifulsoupweb-scraping
Why not just consume the Javascript directly? - Triptych
Duplicate of stackoverflow.com/questions/2148493/… - hymloth
Why you do consume the Javascript directly ? For instance how do you call the JS function JS_Function(var1,var2,var3) from python ? - Kiran

3 Answers

5
votes

Scraping javascript-based webpages is possible with selenium. In particular, try the Selenium WebDriver.

4
votes

I use webkit, which is the browser renderer behind Chrome and Safari. There are Python bindings to webkit through Qt.

And here is a full Python example to execute JavaScript and extract the final HTML.

3
votes

You can use the QtWebKit module of the PyQt4 library