Specflow selenium tests to run against deployed code

Question

I have a question that as yet no one has been able to give me a good answer for.

We have some specflow tests that use selenium to test our web application (SPA), which are included in the build definition for TFS so that they run with the build.

The tests run against a website that has the address specified in the web.config. I think that this means the solution is built and then the tests are run against the url specified in the config file which means its actually testing the last deployed version of the website? Is this right and is there a way of deploying the site and then having the specflow tests run against it?

Whats the best practice here please, thanks.

Answer 1

What I tend to do in this situation is split the build into several steps:

• Build the code and run unit tests
• Deploy the code to a test server
• Run post deployment tests against the new deployment

This ensures that the tests are run against the code I've just checked in but obviously has a few caveats in that you have to make sure that another build doesn't come over a splat your current deployment whilst your tests are still running. The need for mitigation of this will depend on how big a team you have and how much this needs automation, if you can rely on people managiung it manually, great. If not then you can have several test environments, or ideally 1 per dev. This is simplified if you are deploying to the cloud as you can fire up a test deployment per dev easily.

You might also be able to handle some of this with some of the new stuff coming in asp.VNext as you can have the tests self host the web site potentially, removing the need for the deployment at all. Although I would have thought that practically this would only really mean that more of your 'integration' tests could be run this way, you would probably still want some tests to run after deployment, even if they are a subset of smoke tests to check everything still works.