Why are my tests failing when run on the build server?

2
votes

My team is building out a regression suite of SpecFlow tests for one of our websites and have the tests working locally. We've been trying to get the build server (TFS 2013 build server) to be able to run these tests when we check in code as well. We haven't been able to get this to work.

Locally we have our tests running on the NUnit framework, which we thought would work the same on the build server. It seems the build server only wants to run the tests on VS Test Runner which is failing to run our SpecFlow tests every time.

If I log into the build server remotely and run the SpecFlow tests in VS against SpecRun the tests work and pass. We've been playing around with the build definition as well, but haven't had any luck. What needs to be set up to get this to run when code is checked in?

Here is our pattern matching for our SpecFlow test project: enter image description here

And here is an image of our build output as VS Test Runner is trying to execute one of our SpecFlow tests: enter image description here

UPDATE: We've got our tests to run on the build server. However, our tests that are actually trying to hit our web sites keep generating the following error:

Test method AccessTheNDCHomePage threw exception: OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:7055/hub/session/ae10bda6-c46c-4c35-bd9f-f2f9400767ef/url timed out after 60 seconds. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)

After looking around online it seems like this could potentially be because of Firefox update to version 38 and/or a recent Selenium/WebDriver update. Does anyone have any information or tips on fixing this issue?

4
tfsspecflow
Are these UI tests or Unit tests? Are you using the Vsix package for the NUnit test runner or the NuGet package (nuget.org/packages/NUnitTestAdapter)? – jessehouwing
These are UI tests being run with the nuget Nunit package. – Spen
If you are running ui tests, make sure that your build agent is configured to run interactive and that the build agent is logged on, not locked and no Screensaver is enabled. Or, if your tests are using selenium, configure them to use Phantom, which can run without interaction. – jessehouwing
what happens if you visit that url on the server? Is the application running on the server when the test are running? does something in your tfs build deploy the site locally? – Sam Holder
Your question is misleading. Your tests are running, but they're failing. Hook up a debugger and troubleshoot the way you normally would. – Daniel Mann

4 Answers

1
votes

If you are already able, with the same solution, to build and run other nUnit tests with the build then you likley only have a pattern match issue.

The build specified a pattern of assembly names to go find test in. If you change It to included "spec" then your tests will run.

In XAML build It is on the process tab of the build definition editor. In the new build system you select the "VS Test" task and edit It there.

1
votes

Firstly you need to understand what SpecFlow is. It is basically a unit test generation tool. It generates the unit tests in the language defined in the config.

if you wrote plain NUnit tests in your solution what would you need to do to get them to run on TFS? you would have to have a NUnit runner on the the build server and have TFS configured to use this. It's no different for SpecFlow tests.

So the solution? Either generate your tests in MSTest format locally, then when you check these tests in TFS will run them. If you don't want to do that, then set TFS up to be able to run NUnit flavour tests.

EDIT

It sounds as though when you say 'fail to run' the specflow tests, these are not failing, but not being found. Probably you need to adjust either the name of the assemblies that contain your specflow tests so that the current patterns in tfs find the test assemblies, or add filters to allow TFS to find your new assemblies.

0
votes

Here were some of the issues and solutions we found to set up SpecFlow on our build server:

  • The build server wouldn't always recognize our SpecFlow test project - Setting the Platform target of the SpecFlow test to x86 seemed to fix this issue

  • WebDriver time out after 60 seconds error - I am still getting this locally and it still happens when we try to run our tests via the browser on our build server. We were able to run our tests against our Selenium Grid as well as against SauceLabs via the build server. If I find a solution to this I will update my answer.

0
votes

I have had the same issue in the past, the answer is to make sure that your build agent is configured to run interactive.

The build agent must be configured to run interactive in order to run the selenium/coded UI tests against a browser.

https://msdn.microsoft.com/en-us/library/ms181712.aspx#interactive