Random Selenium E2e Tests Fail because of timeouts on Azure DevOps but work locally and with remote Selenium (BrowserStack Automate)

Question

I've got a suite of Selenium tests that work perfectly in my local environment and using Browserstack Automate, but fail on Azure DevOps.

There are no configuration or setting changes when running on Azure Devops.

We've followed all the documentation here: https://docs.microsoft.com/en-us/azure/devops/pipelines/test/continuous-test-selenium?view=vsts

Random tests fail, never the same ones.

The tests always fail because of timeouts. I wait for the pages to load for 5 minutes so it's not a case of the timeouts being too low.

There are no firewalls in place, the application is public.

Authentication always succeeds so the tests are able to load the application.

Not sure what to try next.

Below is a copy of the Azure DevOps log. 4 tests passed but all the other's failed. Usually, only 4-5 tests fail.

This tests works perfectly using BrowserStack Automate (remote selenium) and locally.

2018-11-17T05:40:28.6300135Z  Failed   StripeAdmin_WhenOnTab_DefaultSortIsByIdDescending
2018-11-17T05:40:28.6300461Z Error Message:
2018-11-17T05:40:28.6304198Z  Test method CS.Portal.E2e.Tests.Admin.StripeAdmin.StripeAdminTests.StripeAdmin_WhenOnTab_DefaultSortIsByIdDescending threw exception: 
2018-11-17T05:40:28.6305677Z OpenQA.Selenium.WebDriverTimeoutException: Timed out after 300 seconds
2018-11-17T05:40:28.6307041Z Stack Trace:
2018-11-17T05:40:28.6307166Z     at OpenQA.Selenium.Support.UI.DefaultWait`1.ThrowTimeoutException(String exceptionMessage, Exception lastException)
2018-11-17T05:40:28.6307999Z    at OpenQA.Selenium.Support.UI.DefaultWait`1.Until[TResult](Func`2 condition)
2018-11-17T05:40:28.6308188Z    at CS.Portal.E2e.Tests.Utility.WebDriverUtilities.WaitForElement(IWebDriver driver, By by, Boolean mustBeDisplayed) in D:\a\1\s\CS.Portal.E2e.Tests\Utility\WebDriverUtilities.cs:line 26
2018-11-17T05:40:28.6319651Z    at CS.Portal.E2e.Tests.Admin.StripeAdmin.StripeAdminTests.StripeAdmin_WhenOnTab_DefaultSortIsByIdDescending() in D:\a\1\s\CS.Portal.E2e.Tests\Admin\StripeAdmin\StripeAdminTests.cs:line 51
2018-11-17T05:40:28.6319982Z 
2018-11-17T05:40:34.4671568Z Results File: D:\a\1\s\TestResults\VssAdministrator_factoryvm-az416_2018-11-17_03_08_24.trx
2018-11-17T05:40:34.4692222Z 
2018-11-17T05:40:34.4695222Z Attachments:
2018-11-17T05:40:34.4697610Z   D:\a\1\s\TestResults\672f4d28-5082-42e9-a7e7-f5645aadcfd8\VssAdministrator_factoryvm-az416 2018-11-17 03_02_43.coverage
2018-11-17T05:40:34.4697943Z 
2018-11-17T05:40:34.4698278Z Total tests: 34. Passed: 4. Failed: 30. Skipped: 0.

The timeout occurs only in page loading? or in driver.findElement() as well? — Guy
@jfar Update the question with your code trials and error stack trace — DebanjanB

DebanjanB DebanjanB · Accepted Answer · 2018-11-20T10:30:18

A few lines from your code block would have helped to analyze your issue in a better way.

However, as your tests always fail because of timeouts it is worth to mention that, in general TimeoutException is the outcome of failed ExpectedConditions. However there can be other issues as well.

Some of the approaches to avoid these issues are as follows:

As you mentioned, I wait for the pages to load for 5 minutes... that would be against all the best practices. Instead you need to implement PageLoad, ImplicitWait or WebDriverWait

WARNING: Do not mix implicit and explicit waits. Doing so can cause unpredictable wait times.

You can find a detailed discussion in How can I make sure if some HTML elements are loaded for Selenium
If you are using ChromeDriver and Chrome Browser you must ensure that the binaries are compatible as per the entries below:
- ChromeDriver v2.44 : Supports Chrome v69-71 (same as ChromeDriver 2.43, but with additional bug fixes, released Nov 20, 2018)
- ChromeDriver v2.43 : Supports Chrome v69-71
- ChromeDriver v2.42 : Supports Chrome v68-70
- ChromeDriver v2.41 : Supports Chrome v67-69
Different Browsers renders the HTML DOM differently. So you need to ensure that the Locator Strategies which you are using are optimized.
As per the current WebDriver-W3C Recommendation the following is the list of preferred Locator Strategies:

There is some difference in the performance using CssSelector and XPath. A few take aways:
- For starters there is no dramatic difference in performance between XPath and CSS.
- Traversing the DOM in older browsers like IE8 does not work with CSS but is fine with XPath. And XPath can walk up the DOM (e.g. from child to parent), whereas CSS can only traverse down the DOM (e.g. from parent to child). However not being able to traverse the DOM with CSS in older browsers isn't necessarily a bad thing as it is more of an indicator that your page has poor design and could benefit from some helpful markup.
- An argument in favor of CSS is that they are more readable, brief, and concise while it is a subjective call.
- Ben Burton mentions you should use CSS because that's how applications are built. This makes the tests easier to write, talk about, and have others help maintain.
- Adam Goucher says to adopt a more hybrid approach -- focusing first on IDs, then CSS, and leveraging XPath only when you need it (e.g. walking up the DOM) and that XPath will always be more powerful for advanced locators.
- You can find a detailed discussion in Why should I ever use CSS selectors as opposed to XPath for automated testing?

Conclusion

Keeping the above mentioned factors in consideration you need to implement the Locator Strategy wisely along with the other approaches discussed above which will help you to get rid of the timeouts.

Random Selenium E2e Tests Fail because of timeouts on Azure DevOps but work locally and with remote Selenium (BrowserStack Automate)

2 Answers

Conclusion