SpecFlow wrongly using NUnit

Question

I've just (today) tried SpecFlow for the first time. I'm playing about by creating a new class library in VS2010 Pro and adding a SpecFlow Feature Definition file.

Thing is, the integration doesn't appear to be working properly, with a variety of different errors. I've selected MsTest as the test runner, because I can't be bothered with invoking NUnit (I'd like to use NUnit in the long term but at the moment I just want to get some BDD code working). The generated code files however continue to reference NUnit - which is obviously wrong, since I've just told SpecFlow to run using MsTest. I've done everything I can think of to invoke the code generation again, including creating a brand new class library project with the MsTest option selected in Tools > Options > SpecFlow.

If I leave the test runner field set to 'Auto' and right-click a feature file, then select 'Run SpecFlow Scenarios' I get an error message "Could not find matching test runner".

If I instead change the test runner field to MsTest, I get a different error message on doing the same thing - "Object Reference not set to an instance of an object". I'm not surprised at this one since it's still trying to run NUnit tests even though I've explicitly asked for MsTest, though obviously it shouldn't nullref and present that to the user.

What am I doing wrong? The documentation is not helpful, and as far as I can see, there's no FAQ.

edit #1: I've established that the actual setting I'm looking for is provided using App.Config using the field <unitTestProvider name="MsTest" />. I can see what's happened - the field in the Visual Studio options menu doesn't seem to modify the project you're currently working on. Thing is, this makes it look like that field doesn't do anything at all. I've now persuaded SpecFlow to generate MsTest classes and run using the MSTest runner.

So now the question morphs into a slightly different one: What (if anything) does the Tools > Options > SpecFlow > Test Runner Tool field do?

Answer 1

With VS2010 the correct value is MsTest.2010 not MsTest as documented. Change your app.config (for the test assembly) and it will work fine (at least with SpecFlow 1.8)

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow" />
  </configSections>
  <specFlow>
      <unitTestProvider name="MsTest.2010" />
      <!-- For additional details on SpecFlow configuration options see https://github.com/techtalk/SpecFlow/wiki/Configuration -->
  </specFlow>
</configuration>

Answer 2

In answer to your latest Question. What is the setting "Tools > Options > SpecFlow > Test Runner Tool" this setting controls what will actually run the tests, not what will generate the test code. If it is set to auto i believe it will look at the App.config file where you have set the unitTestProvider to determine what the best tool is to run the tests. An alternaive Test runner made by the same guys as SpecFlow is SpecRun http://www.specrun.com/

So when you go to run the tests it will use this option. As you have discovered though the code generator uses the config file to determine what type of test it should generate (mstest/nunit..)

If you ran the specfow installer ( https://github.com/downloads/techtalk/SpecFlow/SpecFlowSetup_v1.8.1.msi ) to install all the Visual Studio Intergration components when you change the App.config file it normally promps to regenerate the features using the new provider. The manual way to do this though is to right click the Feature and select "Run Custom Tool"

In regards to documentation have you found the git hub wiki? https://github.com/techtalk/SpecFlow/wiki/Documentation

Answer 3

The way I've read this is that the test runner is entirely different to that of the code generator although that doesn't always make sense when the MsTest runner doesn't know about NUnit (I think). Out of the box, the latest version (v2.3.2) even when installed with SpecFlow.MsTest nuget package (of the same version) does not configure your machine to generate MsTest based classes in the background. I am running VS2017 and have Resharper installed as my 'test runner' but the main requirement for generating MsTest based code is a change to the app.config. As per the wiki documentation you also need the following in your app.config. When you save the config you should be prompted for the files to be regenerated.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="specFlow"
             type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow"/>
  </configSections>
  <specFlow>
    <unitTestProvider name="MsTest" />
  </specFlow>
</configuration>

Answer 4

We are using ReSharper as a runner for SpecFlow acceptance tests; it worked well right out of the box. Although ReSharper is not free, but it worth every penny...

Answer 5

I was never able to get SpecFlow working right from Visual Studio, I spent some time working on it but never go anywhere. Though I found these instructions on setting up NUnit in Visual Studio 2010 and I use this shortcut to run my SpecFlow tests with good effect.

Overall we use PowerShell to run a lot of tests and I was able to incorporate the NUnit command line runner and SpecFlow report generator into a single script I can run easily.