Click on a javascript link within python?

15
votes

I am navigating a site using python's mechanize module and having trouble clicking on a javascript link for next page. I did a bit of reading and people suggested I need python-spidermonkey and DOMforms. I managed to get them installed by I am not sure of the syntax to actually click on the link.

I can identify the code on the page as: 

<a href="javascript:__doPostBack('ctl00$MainContent$gvSearchResults','Page$2')">2</a>

Does anyone know how to click on it? or if perhaps there's another tool.

Thanks

3
javascriptpythonscreen-scrapingmechanizespidermonkey
Wouldn't you just click on it normally? If python-spidermonkey and DOMForms are any good it would just work. - Peter C
I was trying to but its really hard to find examples that work. I am actually not sure how to do it. Most of the commands I found in the example didn't work either. I have read of alot of people suggesting these tools for their ability to handle javascript but using the packages is not very straight forward. - Lostsoul
If I need to deal with JavaScript, I avoid mechanize (or twill, which I prefer) and instead use something like Selenium or Splinter (which is my favorite between the two). - brandizzi

3 Answers

6
votes

I mainly use HtmlUnit under jython for these use cases. Also I published a simple article on the subject: Web Scraping Ajax and Javascript sites.

2
votes

instead of struggling with python-spidermonkey try webkit's qt python bindings.

Here is a full example to execute JavaScript and extract the final HTML.

0
votes

how about calling __doPostBack('ctl00$MainContent$gvSearchResults','Page$'+pageid); (javascript method, via python-spidermonkey)