pageTimings not getting captured with Selenium and Browsermob proxy

1
votes

I am using Browsermob proxy with selenium webdriver to capture HAR log. I am able to generate logs but the pageTimings are not getting captured. Rest all data is getting captured. Any idea what is wrong here?

Do I need to use some wait using this call:

PUT /proxy/[port]/wait - wait till all request are being made

enter image description here Updates after following the short answer :

  1. Taken latest snapshot from git , built it.
  2. Started the proxy from distribution module - 'Main' programme.
  3. Logs getting captured
  4. After each page source change , I am making a call to force the proxy to end the page by PUTing to /proxy/{port}/har/pageRef
  5. Logs got generated with page references but still the pageTimings are not getting populated.

Updated screenshot:

enter image description here

Error in log after using little proxy:

[ERROR 2015-05-18T10:29:00,288 org.littleshoot.proxy.impl.ClientToProxyConnection] (LittleProxy-ClientToProxyWorker-1) (AWAITING_INITIAL) [id: 0x1961ff09, /0:0:0:0:0:0:0:1:63598 => /0:0:0:0:0:0:0:1:8445]: Caught an exception on ClientToProxyConnection java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:1.8.0_45]
    at sun.nio.ch.SocketDispatcher.read(Unknown Source) ~[?:1.8.0_45]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) ~[?:1.8.0_45]
    at sun.nio.ch.IOUtil.read(Unknown Source) ~[?:1.8.0_45]
    at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[?:1.8.0_45]
    at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:447) ~[netty-all-4.0.27.Final.jar:4.0.27.Final]
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) ~[netty-all-4.0.27.Final.jar:4.0.27.Final]
    at io.netty.buffer.WrappedByteBuf.writeBytes(WrappedByteBuf.java:641) ~[netty-all-4.0.27.Final.jar:4.0.27.Final]
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241) ~[netty-all-4.0.27.Final.jar:4.0.27.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) [netty-all-4.0.27.Final.jar:4.0.27.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-all-4.0.27.Final.jar:4.0.27.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-all-4.0.27.Final.jar:4.0.27.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-all-4.0.27.Final.jar:4.0.27.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-all-4.0.27.Final.jar:4.0.27.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-all-4.0.27.Final.jar:4.0.27.Final]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_45]
2
browsermob
The error message in the log states that the remote host closed the connection. Do you see this error when you're not using BMP? – Jason Hoetger

2 Answers

1
votes

Short answer: If you are using the 2.1.0-beta-1 or higher (see the github page), you can force the proxy to end a page by PUTing to /proxy/{port}/har/pageRef . This will populate the pageTimings object in the HAR. (It will also start a new page.)

Update: The "creator" section of the HAR should indicate the littleproxy module is being used:

"log":{
   "version":"1.2",
   "creator":{
      "name":"BrowserMob Proxy",
      "version":"2.1.0-beta-1-littleproxy",
      "comment":""
   },

Here is an example of a populated pageTimings section after ending the first page:

  "pages":[
     {
        "id":"Page 0",
        "startedDateTime":"2015-05-16T12:37:48.406-07:00",
        "title":"Page 0",
        "pageTimings":{
           "onLoad":89648,
           "comment":""
        },
        "comment":""
     },
     {
        "id":"Page 1",
        "startedDateTime":"2015-05-16T12:39:18.054-07:00",
        "title":"Page 1",
        "pageTimings":{
           "comment":""
        },
        "comment":""
     }
  ],

Long answer: Unlike a web browser, BrowserMob Proxy doesn't actually know what a "page" is. It only sees individual requests and responses, so you need to explicitly tell BMP when a page begins and ends. The Java interface provides the newPage() method, and the REST API provides the /proxy/{port}/har/pageRef endpoint. However, since BMP is just a proxy, requests could come from multiple browsers/clients for multiple pages at the same time, and BMP would not have any way of knowing which request belongs to which page. Depending on your use case, page timing information may not be meaningful.

0
votes

It works for me:

BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(0);
HashSet<CaptureType> enable = new HashSet<CaptureType>();
enable.add(CaptureType.REQUEST_HEADERS);
enable.add(CaptureType.REQUEST_CONTENT);
enable.add(CaptureType.RESPONSE_HEADERS);
proxy.enableHarCaptureTypes(enable);
HashSet<CaptureType> disable = new HashSet<CaptureType>();
disable.add(CaptureType.REQUEST_COOKIES);
disable.add(CaptureType.RESPONSE_COOKIES);
proxy.disableHarCaptureTypes(disable);

//get the Selenium proxy object
Proxy selProxy = ClientUtil.createSeleniumProxy(proxy);

capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, selProxy);
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS,true);

WebDriver driver = new FirefoxDriver(new FirefoxBinary(),profile,capabilities);


driver.get(url);

Har har = proxy.getHar();