3
votes

I want to use selenium to scrape off some website. I can't access the website via my own internet connection, so I need to use browsec mozilla addon for that.

I am unable to launch firefox with selenium with the add-on enabled. Here is what I have tried:

import selenium
from selenium import webdriver

url = "http://url"
profile = webdriver.FirefoxProfile()
profile.add_extension('[email protected]')
#profile.add_extension("C:\Users\urs\AppData\Roaming\Mozilla\Firefox\Profiles\abc.default\extensions\[email protected]")
driver = webdriver.Firefox(firefox_profile=profile)

if __name__ == "__main__":
   driver.get(url)
   driver.wait(5)
   driver.quit()

I have tried putting the extension in the same directory where my script is and using the following

profile.add_extension('[email protected]')

which gives me this error when I run:

Traceback (most recent call last): File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile .py", line 346, in _addon_details with open(os.path.join(addon_path, 'install.rdf'), 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\Usr\AppD ata\Local\Temp\[email protected]\install.rdf'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "test.py", line 7, in profile.add_extension("[email protected]") File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile .py", line 95, in add_extension self._install_extension(extension) File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile .py", line 274, in _install_extension addon_details = self._addon_details(addon) File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile .py", line 351, in _addon_details raise AddonFormatError(str(e), sys.exc_info()[2]) selenium.webdriver.firefox.firefox_profile.AddonFormatError: ("[Errno 2] No such file or directory: 'C:\\Users\\Usr\\AppData\\Local\\Temp\\tmp0hn [email protected]\\install.rdf'", )

I also tried giving the path to the extension:

profile.add_extension("C:\Users\urs\AppData\Roaming\Mozilla\Firefox\Profiles\abc.default\extensions\[email protected]")

And I ran into this error:

profile.add_extension("C:\Users\Hassan\AppData\Roaming\Mozilla\Firefox\Profi les\n5jwlj9l.default\extensions\[email protected]") ^ SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in positio n 2-3: truncated \UXXXXXXXX escape

Formatting the path string like below doesn't help either.

profile.add_extension(r"C:\Users\urs\AppData\Roaming\Mozilla\Firefox\Profiles\abc.default\extensions\[email protected]")

I get the following:

Traceback (most recent call last): File "test.py", line 7, in profile.add_extension(r"C:\Users\Hassan\AppData\Roaming\Mozilla\Firefox\Prof iles\n5jwlj9l.default\extensions\[email protected]") File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile .py", line 95, in add_extension self._install_extension(extension) File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile .py", line 274, in _install_extension addon_details = self._addon_details(addon) File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile .py", line 351, in _addon_details raise AddonFormatError(str(e), sys.exc_info()[2]) selenium.webdriver.firefox.firefox_profile.AddonFormatError: ("[Errno 2] No such file or directory: 'C:\\Users\\usr\\AppData\\Local\\Temp\\tmp1he [email protected]\\install.rdf'", )

How do I configure selenium to run firefox with browsec enabled by default?

3

3 Answers

3
votes

I found this article rather helpful.

Instead of adding the extension to the profile, you install it after the browser has been created:

from selenium import webdriver


driver = webdriver.Firefox()
# This installs adblock plus
driver.install_addon("/home/your_username/coding/Project/seleniumTest/adblock.xpi", temporary=True)
driver.get('https://www.stackoverflow.com')

Be sure to add the .xpi to your project folder!

1
votes

You can try to create profile on firefox browser like - On windows Run --> type

"firefox.exe -P" 

It will open profile manager. Create new profile. Start firefox from that profile, add plugins. And use that same profile with code..Sometime it worked for me..

1
votes

Sorry for my English))

Most likely you are using the new version of Firefox (Quantum - from the 57th version inclusive). In newer versions of Firefox, the extension metadata is not stored in the install.rdf file, but in the manifest.json file. Selenium does not know this yet (in version 3.11, and learns only in 3.14). Therefore, when trying to connect an extension, it looks for habit install.rdf.

Here the author wrote a class that slightly changes the connection function of the extension, and instead of install.rdf, selenium looks for metadata in manifest.json.

What you need to do:

# Add Import

import json
import os
import sys

from selenium.webdriver.firefox.firefox_profile import AddonFormatError

# Add class

class FirefoxProfileWithWebExtensionSupport(webdriver.FirefoxProfile):
    def _addon_details(self, addon_path):
        try:
            return super()._addon_details(addon_path)
        except AddonFormatError:
            try:
                with open(os.path.join(addon_path, 'manifest.json'), 'r') as f:
                    manifest = json.load(f)
                    return {
                        'id': manifest['applications']['gecko']['id'],
                        'version': manifest['version'],
                        'name': manifest['name'],
                        'unpack': False,
                    }
            except (IOError, KeyError) as e:
                raise AddonFormatError(str(e), sys.exc_info()[2])

# Declare Firefox_profile written class

profile = FirefoxProfileWithWebExtensionSupport()

Further as usual)))

Good luck)))