Multi-client remote debugging using ChromeDriver and Chrome DevTools protocol

Question

So with Chrome 63 there is now support for Multi-client remote debugging (https://developers.google.com/web/updates/2017/10/devtools-release-notes)

What I want to achieve is use the Chrome DevTools Protocol HeapProfiler with some selenium tests. I'm running version 64 Chrome dev channel and ChromeDriver 2.33.

ChromeOptions options = newChromeOptions();
options.addArguments("--remote-debugging-port=9222");
WebDriver driver = new ChromeDriver(options);
... selenium stuff

A new chrome window will open and hang until it times out. I can confirm that the chrome window opened is chrome 64 by going to help > about google chrome to check the version. I get this error which appears to be the the webdriver losing connection.

Exception in thread "main" org.openqa.selenium.WebDriverException: chrome not 
reachable

The DevTools Protocol is working because I am able to open http://localhost:9222 in another chrome window and see debugging interface.

Has anyone been able to get these two things to work together?

Thanks :)

Answer 1

Here the catch was that if you pass the "remote-debugging-port" switch then chromedriver has a bug where it still internally assigns a randon port and keep trying to connect to it rather than connecting to 9222 port.

options.addArguments("--remote-debugging-port=9222");

We can solve this by skipping this command switch and let chrome decides this random port and extract this port number from chromedriver logs.

I made it work and here I have blogged it in detail.

https://medium.com/@sahajamit/selenium-chrome-dev-tools-makes-a-perfect-browser-automation-recipe-c35c7f6a2360

Answer 2

Selenium 4 release will have a user friendly API for Chrome DevTools protocol. I just finished implementing Network and Performance domains for the Selenium Java client. https://github.com/SeleniumHQ/selenium/pull/7212

In addition, there is a generic API for all domains in Java client that was merged a while ago. All those new features will be released probably in the next Alpha release.

This is a nice article on how to use Log: https://codoid.com/selenium-4-chrome-devtools-log-entry-listeners/

Answer 3

Here is what i do to get the information needed fro remotedebugging nd additionally to prevent defining the port. I get it through the SeleniumLog-API

DesiredCapabilities capabilities = DesiredCapabilities.chrome();
ChromeOptions options = new ChromeOptions();
options.setBinary(chromeBin);
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
LoggingPreferences logPref = new LoggingPreferences();
logPref.enable(LogType.DRIVER, Level.ALL);
driverInstance = new ChromeDriver(capabilities);

LogEntries x = driverInstance.manage().logs().get(LogType.DRIVER);
    for(LogEntry e:x.getAll()){
        if(e.getMessage().contains("DevTools request:")){
            String url = e.getMessage().replaceFirst("DevTools request:", "").trim();
        }

        if(e.getMessage().contains("DevTools response:")){
            String json = e.getMessage().replaceFirst("DevTools response:", "");
            try {
                if("page".equals(JSONUtil.get(json,"type" ))){
                    webSocketDebuggerUrl = JSONUtil.get(json,"webSocketDebuggerUrl" );
                }
            } catch (Exception e1) {
                e1.printStackTrace();
            }

        }
        System.out.println(e.getMessage());
    }

The JSONUtil i use is my own tool, so don't wonder, just replace with whatever code to extract from the jsontext.

Answer 4

Here's a fairly robust implementation in java using the same target tab with selenium 3.13 & cdp4j 3.0.2-SNAPSHOT. Easily translates to any language.

package com.company;

import com.google.gson.JsonElement;
import com.google.gson.JsonParser;
import io.webfolder.cdp.session.SessionFactory;
import org.openqa.selenium.HasCapabilities;
import org.openqa.selenium.Keys;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeDriverService;

import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Map;

public class Main {

    public static void main(String[] args) {

        System.setProperty(ChromeDriverService.CHROME_DRIVER_EXE_PROPERTY, "C:\\path\\to\\chromedriver.exe");
        var driver = new ChromeDriver();

        try {
            var cdp = findCdpEndpoint(driver);
            System.out.println(cdp.toString());
            try (var factory = new SessionFactory(cdp.getPort())) {
                driver.navigate().to("https://google.com");
                String seTargetId = getSeTargetId(cdp, driver.getTitle());

                try (var session = factory.connect(seTargetId)) {
                    session.waitDocumentReady();
                    session.sendKeys("Astronauts");
                    driver.getKeyboard().sendKeys(Keys.RETURN);
                    session.wait(2000);
                    driver.navigate().to("http://www.google.com");
                    session.waitDocumentReady();
                }
            }
        } catch (Exception ex) {
            System.out.println(ex.toString());
        }

        driver.quit();
    }

    private static String getSeTargetId(URL cdp, String title) throws IOException {
        for (JsonElement element : new JsonParser().parse(new InputStreamReader(cdp.openStream(), "UTF-8")).getAsJsonArray()) {
            var object = element.getAsJsonObject();
            if (title == null || title.isEmpty()
                    ? object.get("type").getAsString().equalsIgnoreCase("page")
                    : object.get("title").getAsString().equalsIgnoreCase(title)) {
                return object.get("id").getAsString();
            }
        }
        throw new IllegalStateException("Selenium target not found.");
    }

    private static URL findCdpEndpoint(WebDriver driver) throws IOException {
        var capChrome = (Map<?,?>) ((HasCapabilities)driver).getCapabilities().getCapability("chrome");
        var userDataDir = (String) capChrome.get("userDataDir");
        var port = Integer.parseInt(Files.readAllLines(Paths.get(userDataDir, "DevToolsActivePort")).get(0));
        return new URL("http", "localhost", port, "/json");
    }
}