How can I get table with dynamic data from a url using Selenium and xpath, or other alternative?

1
votes

I am trying to retrieve the main "Network Status" table data from this site:

http://dev.cryptolions.io/mainnet/

I have tried many combinations of selenium functions populated with Chrome's copy selector and copy Xpath, but am not having any luck. The site opens with selenium, but I keep getting errors similar to this. What am I missing here?

Getting the table in any number of data structures would be ok. But a dictionary, JSON, or something I can load into a pandas df would be ideal.

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="bpTableSystem"]"}
(Session info: chrome=67.0.3396.87) (Driver info: chromedriver=2.40.565498 (ea082db3280dd6843ebfb08a625e3eb905c4f5ab),platform=Windows NT 10.0.17134 x86_64)

My Code:

driver = webdriver.Chrome(r'C:\chromedriver_win32\chromedriver.exe') 
browser = webdriver.Chrome()

browser.get('http://dev.cryptolions.io/mainnet/')

time.sleep(30)
content = driver.find_element_by_xpath('//*[@id="bpTableSystem"]')
print content.text
1
pythonseleniumselenium-webdriverxpath

1 Answers

1
votes

The problem is that you've defined two Chrome instances: browser and driver. You navigate to target page with browser while trying to find table with driver.

Simply replace

browser.get('http://dev.cryptolions.io/mainnet/')

with

driver.get('http://dev.cryptolions.io/mainnet/')