SpecFlow BDD UI tests running on Azure DevOps pipeline

Question

I have created a test suite using SpecFlow BDD with Selenium that I am trying to run on the Azure DevOps using Pipelines:

trigger:
  - master
pool:
  name: Hosted Windows 2019 with VS2019
  vmImage: windows-latest
schedules:
  - cron: '00 16 * * MON,TUE,WED,THU,FRI'
    displayName: Weekdays 4PM build
    branches:
      include:
        - master
        - releases/*
      exclude:
        - development
        - feature/*
jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: windows-latest
    steps:
      - task: PowerShell@2
        displayName: Clone the pipelines repository
        inputs:
          targetType: inline
          script: >
            git clone -b master
            https://azuredevops:$($env:token)@dev.azure.com/abc-comapny/MyAzure/_git/MyProject.Builder
            pipeline-repo/
        env:
          token: $(System.AccessToken)
      - task: Bash@3
        displayName: Prepare the nuget.config file
        inputs:
          targetType: inline
          script: |
            echo copying the nuget.config to the root folder
            cp ./pipeline-repo/templates/nuget.config .
      - task: NuGetCommand@2
        displayName: Authenticate with Azure DevOps NuGet
        inputs:
          command: custom
          arguments: >-
            sources update -Name "Azure DevOps" -Username "vsts" -Password
            "$(System.AccessToken)" -StorePasswordInClearText -ConfigFile
            ./nuget.config
      - task: UseDotNet@2
        displayName: Using SDK Version (3.1.x)
        inputs:
          version: 3.1.x
      - task: DotNetCoreCLI@2
        displayName: Restore
        inputs:
          command: restore
          projects: '**/*.sln'
      - task: DotNetCoreCLI@2
        displayName: Build the solution
        inputs:
          command: build
          projects: '**/*.sln'
      - task: SpecFlowPlus@0
        inputs:
          projectFilePath: .
          projectName: MyProject.Qa
          projectLanguage: en
          workItemPrefix: AUTO
      - task: DotNetCoreCLI@2
        displayName: Publish
        inputs:
          command: publish
          publishWebProjects: false
          projects: '**/*.sln'
          arguments: >-
            --configuration Release -o
            $(build.artifactstagingdirectory)/SeleniumTests
          zipAfterPublish: false
          modifyOutputPath: false
      - task: PublishBuildArtifacts@1
        displayName: Publish Artifact
        inputs:
          PathtoPublish: $(build.artifactstagingdirectory)
        condition: succeededOrFailed()
      - script: >-
          dotnet test MyProject.Qa.sln -r "$(Agent.TempDirectory)/" --logger
          "trx;LogFilename=testresults.trx" 
        displayName: Run Unit Tests and Code Coverage
        continueOnError: false

I have tried looking up the instructions given at:

https://swimburger.net/blog/dotnet/how-to-run-net-core-selenium-ui-tests-on-azure-devops-pipelines

https://www.stuartwhiteford.com/running-selenium-ui-tests-in-an-azure-devops-pipeline/

https://azuredevopslabs.com/labs/vstsextend/selenium/#:~:text=Navigate%20to%20Pipelines%20under%20Pipelines,successful%2C%20release%20will%20be%20triggered.

The steps are all passing, but I do not get any result or see any output for the test results. I have 2 projects, both MS Test, one for the BDD Specifications and the other for the Selenium framework files:

• MyProject.Qa.Specification.csproj

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  
    <IsPackable>false</IsPackable>
  </PropertyGroup>
  
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.1.2" />
    <PackageReference Include="MSTest.TestFramework" Version="2.1.2" />
    <PackageReference Include="coverlet.collector" Version="1.3.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
    <PackageReference Include="Selenium.Support" Version="3.141.0" />
    <PackageReference Include="Selenium.WebDriver" Version="3.141.0" />
    <PackageReference Include="Selenium.WebDriver.ChromeDriver" Version="85.0.4183.8700" />
    <PackageReference Include="Selenium.WebDriver.GeckoDriver" Version="0.27.0" />
    <PackageReference Include="SpecFlow" Version="3.4.3" />
    <PackageReference Include="SpecFlow.Assist.Dynamic" Version="1.4.2" />
    <PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.4.3" />
    <PackageReference Include="SpecRun.Runner" Version="3.4.12" />
    <PackageReference Include="SpecRun.SpecFlow" Version="3.4.12" />
  </ItemGroup>
  
  <ItemGroup>
    <ProjectReference Include="..\MyProject.Qa.Framework\MyProject.Qa.Framework.csproj" />
  </ItemGroup>
  
  <ItemGroup>
  <Folder Include="FeatureFiles\" />
  <Folder Include="Hooks\" />
  <Folder Include="SpecFlowPlusRunner" />
  <Folder Include="StepDefinitions\" />
  </ItemGroup>
  

• MyProject.Qa.Framework.csproj:

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  
    <IsPackable>false</IsPackable>
  </PropertyGroup>
  
  <ItemGroup>
    <PackageReference Include="DotNetSeleniumExtras.PageObjects" Version="3.11.0" />
    <PackageReference Include="DotNetSeleniumExtras.WaitHelpers" Version="3.11.0" />
    <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
    <PackageReference Include="NUnit" Version="3.12.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.17.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />
    <PackageReference Include="Selenium.Support" Version="3.141.0" />
    <PackageReference Include="Selenium.WebDriver" Version="3.141.0" />
    <PackageReference Include="Selenium.WebDriver.ChromeDriver" Version="85.0.4183.8700" />
    <PackageReference Include="Selenium.WebDriver.GeckoDriver" Version="0.27.0" />
    <PackageReference Include="Selenium.WebDriver.GeckoDriver.Win64" Version="0.27.0" />
    <PackageReference Include="SpecFlow" Version="3.4.3" />
    <PackageReference Include="System.Configuration.ConfigurationManager" Version="4.7.0" />
  </ItemGroup>
  
  <ItemGroup>
    <Folder Include="CommonStructures\" />
    <Folder Include="Controllers\" />
    <Folder Include="Data\" />
    <Folder Include="Dialogs\" />
    <Folder Include="Extensions\" />
    <Folder Include="Pages\" />
  </ItemGroup>
  

Has anyone used SpecFlow BDD UI tests with Azure DevOps Pipeline to run the UI tests in a non-headless browser type run? If so, could you inform, what you did and if I am doing anything wrong with the above 'azure-pipeline.yaml' file and need to change anything to run these tests?

Also, how do I use the above to send across a mail to all the stakeholders? Can I do that with the YAML or do I need to do that from some other part of the Azure DevOps, once I have the report up?

Answer 1

I think you are missing a task to publish the test results to Azure DevOps. You are executing dotnet test and get a trx- file (the result file), but you are not doing anything with it.

You need to use the Publish Test Result task (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml) afterward to upload the trx- file to Azure DevOps.

Or you change the dotnet test command to use the Visual Studio Test task (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/vstest?view=azure-devops). This tasks makes the upload of the trx- file automatically.