Browsermob proxy with selenium generates empty output

Question

I am using selenium 3.0.2 and browsermob proxy 0.7.1 to capture the network data. All I am getting is an empty JSON. My code is:

server = Server("/Users/dev/Downloads/browsermob-proxy-2.1.2/bin/browsermob-proxy")
server.start()
proxy = server.create_proxy()

if browser is None:
    profile = webdriver.FirefoxProfile()
    profile.set_preference("browser.privatebrowsing.autostart", True)

browser = webdriver.Firefox(firefox_profile=profile, proxy=proxy.selenium_proxy())
proxy.new_har("google", options={'captureHeaders': True, 'captureContent': True})

browser.get("https://google.com/")

print(proxy.har)

All I get is this empty JSON

{'log': {'pages': [{'id': 'google', 'comment': '', 'pageTimings': {'comment': ''}, 'startedDateTime': '2016-12-01T14:23:24.984-05:00', 'title': 'google'}], 'entries': [], 'version': '1.2', 'creator': {'comment': '', 'name': 'BrowserMob Proxy', 'version': '2.1.2'}, 'comment': ''}}

I would also really like to know the answer to this question! The HAR is empty, I think, because the proxy isn't actually being used. For http (rather than https) sites, I can get the code to work by using: profile.set_proxy(proxy.selenium_proxy()) in place of passing the proxy object in, but the set_proxy method is deprecated... For HTTPS, I think this issue is also relevant: bugzilla.mozilla.org/show_bug.cgi?id=1103196 — user3468054
Is the page loading successfully in the browser? Since you're getting an empty HAR, that means the proxy is started, but the proxy hasn't seen any requests. — Jason Hoetger

Jason Hoetger Jason Hoetger · Accepted Answer · 2016-12-29T06:00:52

I suspect this is caused by the same GeckoDriver bug/missing feature in this question: GeckoDriver does not currently support the "proxy" desired capability that ChromeDriver and the old FirefoxDriver supported.

Until GeckoDriver/Marionette is updated, you can work around this problem by setting the proxy values on the Firefox Profile directly. The relevant proxy fields on the profile are:

network.proxy.http
network.proxy.http_port
network.proxy.ssl
network.proxy.ssl_port
network.proxy.type (set to 1, "manual")

BrowserMob Proxy has a test that shows how to do this in Java. It should be easy to translate that into a Python equivalent, since you already have a firefox_profile object.

Browsermob proxy with selenium generates empty output

5 Answers