python - How can we use Selenium Webdriver in colab.research.google.com?

35
votes

I want to use Selenium Webdriver of Chrome in colab.research.google.com for fast processing. I was able to install Selenium using !pip install selenium but the webdriver of chrome needs a path to webdriverChrome.exe. How am I suppose to use it?

P.S.- colab.research.google.com is an online platform which provides GPU for fast computational problems related to deep learning. Please refrain from solutions such as webdriver.Chrome(path).

4
pythonseleniumselenium-webdrivergoogle-colaboratory
I think I mentioned "colab.research.google.com". I know how webdriver works on a local machine. But as colab research google is an online platform which provides GPU for fast machine learning processing problems, I want to use webdrive on this above mentioned online platform. - john mich
A same problem is in this link: stackoverflow.com/questions/54327654/… - hamed baziyad
seems like it was asked 7 days ago - john mich
@johnmich did you solve the problem? - Dimanjan
@Dimanjan Hey, I have stopped trying this. Use-case was scrapped and so did not explored further. - john mich

4 Answers

65
votes

You can do it by installing the chromium webdriver and adjusting some options such that it does not crash in google colab:

!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.get("https://www.webite-url.com")
14
votes

I made my own library to make it easy.

!pip install kora -q
from kora.selenium import wd
wd.get("https://www.website.com")
12
votes

this one worked in colab

!pip install selenium
!apt-get update 
!apt install chromium-chromedriver

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
driver =webdriver.Chrome('chromedriver',chrome_options=chrome_options)
-2
votes

You can can rid of using .exe file by using WebDriverManager so instead of this

System.setProperty("webdriver.gecko.driver", "driverpath/.exe");
WebDriver driver = new FirefoxDriver();

you will be writing this

WebDriverManager.firefoxdriver().setup();
WebDriver driver = new FirefoxDriver();

All you need is add the dependecy to the POM file(Im assuming you using maven or some build tool) Please see my full answer about how to use this in this link Using WebdriverManager