4
votes

I'm trying to scrape some stuff using Selenium/headless chrome. In some pages, the driver will not move to the next step until the full page is loaded (even though the relevant elements are present).

I tried setting the pageLoadStrategy capability like so:

caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {"args" => [ "--headless" ]})
      caps['pageLoadStrategy']='eager'
self.driver||=Selenium::WebDriver.for :chrome, :desired_capabilities =>  caps

I get the following error:

Selenium::WebDriver::Error::UnknownError: unknown error: cannot parse capability: pageLoadStrategy
from unknown error: page load strategy unsupported

Is there a way to make this work with chrome?

According to this post, pageLoadStrategy is supported, but I can't get the 'eager' option to work.

Page load strategy for Chrome driver

If not, are there more tried and true ways of doing this. I've been having a lot of trouble with Chrome (it also doesn't support unexpectedAlertBehaviour)

5
The eager strategy is not yet supported (source). Use none and wait for document.readyState to be interactive. - Florent B.
Thanks @FlorentB. Whats the best way to implement that wait (document.readyState interactive) . Also, looking at the source code I see that unexpectedAlerBehaviour is implemented. I tried this extensively like 3 weeks ago with no luck. Any thoughts on how to make that work? I should prob post a new question with that. - Gabriel

5 Answers

3
votes

Chrome does not support PageLoadStrategy.EAGER. But you can try PageLoadStrategy.NONE. Sometimes you need to synchronize driver wait for some elemnts but possibility is less.

2
votes

ChromeDriver 77.0 (which supports Chrome version 77) now supports eager as pageLoadStrategy.

Resolved issue 1902: Support eager page load strategy [Pri-2]

See release notes

1
votes

For those who are waiting for the "eager" feature. You can check the "CheckSupport" function in the source code.

It seems it is still not supported even for now.

0
votes

The eager strategy now seems to be supported by chrome since v66 (source).