python app with selenium failes with MaxRetryError

0
votes

i am trying to host a python app in docker

i am running selenium standalone chrome in docker and i can connect to it running my python app locally.

my application looks like this:

def web_scrape():
url = "https://who.maps.arcgis.com/apps/opsdashboard/index.html#/ead3c6475654481ca51c248d52ab9c61"
#setup webdriver
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--headless')
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--remote-debugging-port=9222")
driver = webdriver.Remote(command_executor="http://localhost:4444/wd/hub",                            desired_capabilities=options.to_capabilities())
driver.get(url)
time.sleep(20)
html = driver.execute_script("return document.documentElement.outerHTML")
#Use BeautifulSoup for working with html
soup = BeautifulSoup(html, "html.parser")
covid_soup = soup.find("div", id="ember44").div.nav.find_all("span", class_="flex-horizontal")
covid_dict = {}
for i in covid_soup:
    country = i.find("strong").get_text(strip=True)
    country = clean_country_name(country)
    imgURL = i.p.find_next("p").find_next("p").find("img").get('src') 
    color = get_covid_color(imgURL)
    covid_dict[country] = color
save_to_json(covid_dict)

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4444): Max retries exceeded with url: /wd/hub/session (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f44c2ac6c40>: Failed to establish a new connection: [Errno 111] Connection refused'))

does anyone have any suggestions on what could be wrong?

2
pythondockerselenium-webdriverselenium-chromedriver

2 Answers

0
votes

how have you setup your testing environment in docker, plus you could replace localhost with selenium-hub at the command_executor argument in the meantime

0
votes

Had the same issue. Tests running from Docker container were failing to drive Chrome using Selenium running in Docker container.

The issue was that Selenium server was not available (while browser container was already up) when tests started to run.

Try calling below function before running actual tests. It will make sure the server is available.

def test_selenium_server_available():
    import requests
    from requests.adapters import HTTPAdapter
    from requests.packages.urllib3.util.retry import Retry

    session = requests.Session()
    retry = Retry(connect=5, backoff_factor=0.5)
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)

    session.get("http://localhost:4444/wd/hub")