I have a simple ASP.Net (MVC) application with a Unit test project that I have setup in Azure DevOps.
The unit test project has a chutzpah.json file and a file with four jasmine.js tests. These tests test files in the main ASP.Net project. I have the Chutzpah Test Adapter for the Test Explorer installed and the tests appear, run and pass from within the Test Explorer within Visual Studio 2017.
In DevOps, similar to this blog, I added a second Visual Studio Test task to my build pipeline (a default ASP.Net template) with the Test files to **\chutzpah.json. This task runs, the log shows the tests passing, then states that the "Test Run Failed". This makes the task and therefore the build fail.
The output is
2018-11-08T11:52:17.5347601Z ##[section]Starting: VsTest - Front End
2018-11-08T11:52:17.5354696Z ==============================================================================
2018-11-08T11:52:17.5354757Z Task : Visual Studio Test
2018-11-08T11:52:17.5354793Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2018-11-08T11:52:17.5354871Z Version : 2.142.12
2018-11-08T11:52:17.5354908Z Author : Microsoft Corporation
2018-11-08T11:52:17.5354962Z Help : [More information](https://go.microsoft.com/fwlink/?LinkId=835764)
2018-11-08T11:52:17.5354999Z ==============================================================================
2018-11-08T11:52:18.2871153Z SystemVssConnection exists true
2018-11-08T11:52:18.3290569Z SystemVssConnection exists true
2018-11-08T11:52:18.4136045Z Running tests using vstest.console.exe runner.
2018-11-08T11:52:18.4136143Z ======================================================
2018-11-08T11:52:18.4136511Z Test selector : Test assemblies
2018-11-08T11:52:18.4136597Z Test filter criteria : null
2018-11-08T11:52:18.4136924Z Search folder : D:\a\1\s
2018-11-08T11:52:18.4137559Z VisualStudio version selected for test execution : latest
2018-11-08T11:52:18.4259721Z Run in parallel : false
2018-11-08T11:52:18.4307979Z Run in isolation : false
2018-11-08T11:52:18.4309258Z Path to custom adapters : null
2018-11-08T11:52:18.4310180Z Other console options : null
2018-11-08T11:52:18.4310482Z Code coverage enabled : false
2018-11-08T11:52:18.4310627Z Diagnostics enabled : false
2018-11-08T11:52:18.4311321Z SystemVssConnection exists true
2018-11-08T11:52:18.4334452Z Run the tests locally using vstest.console.exe
2018-11-08T11:52:18.4334531Z ========================================================
2018-11-08T11:52:18.4347193Z Test selector : Test assemblies
2018-11-08T11:52:18.4351418Z Test assemblies : **\chutzpah.json
2018-11-08T11:52:18.4351642Z Test filter criteria : null
2018-11-08T11:52:18.4355360Z Search folder : D:\a\1\s
2018-11-08T11:52:18.4357017Z Run settings file : D:\a\1\s
2018-11-08T11:52:18.4358825Z Run in parallel : false
2018-11-08T11:52:18.4360300Z Run in isolation : false
2018-11-08T11:52:18.4382185Z Path to custom adapters : null
2018-11-08T11:52:18.4382534Z Other console options : null
2018-11-08T11:52:18.4384839Z Code coverage enabled : false
2018-11-08T11:52:18.4386257Z Diagnostics enabled : false
2018-11-08T11:52:18.4388538Z Rerun failed tests: false
2018-11-08T11:52:18.4391303Z VisualStudio version selected for test execution : latest
2018-11-08T11:52:18.6093014Z ========================================================
2018-11-08T11:52:19.1694099Z ======================================================
2018-11-08T11:52:20.3093149Z [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" @D:\a\_temp\be5d5f11-e34c-11e8-a8a9-2d59e72db6fd.txt
2018-11-08T11:52:20.4991286Z Microsoft (R) Test Execution Command Line Tool Version 15.8.0
2018-11-08T11:52:20.4991453Z Copyright (c) Microsoft Corporation. All rights reserved.
2018-11-08T11:52:20.4991515Z
2018-11-08T11:52:20.5016812Z vstest.console.exe
2018-11-08T11:52:20.5016907Z "D:\a\1\s\TestMVCApplication.Tests\chutzpah.json"
2018-11-08T11:52:20.5016952Z /logger:"trx"
2018-11-08T11:52:20.5017011Z /TestAdapterPath:"D:\a\1\s"
2018-11-08T11:52:20.7387852Z Starting test execution, please wait...
2018-11-08T11:52:27.4042414Z Error: DEPRECATION: The catchExceptions option is deprecated and will be replaced with stopOnSpecFailure in Jasmine 3.0
2018-11-08T11:52:27.4042924Z [{file:file:///D:/a/1/s/packages/Chutzpah.4.4.2/tools/ChutzpahTestHarnessFiles/jasmine/v2/jasmine.js,line:949,function:deprecated}]
2018-11-08T11:52:27.4044557Z While Running:D:\a\1\s\TestMVCApplication.Tests\js-tests\utilitiy-tests.js
2018-11-08T11:52:27.4044678Z
2018-11-08T11:52:27.4767512Z Passed app-funcs: add() Correctly adds 3 numbers
2018-11-08T11:52:27.4768451Z Passed app-funcs: add() Correctly adds 2 numbers
2018-11-08T11:52:27.4768908Z Passed app-funcs: add() Correctly adds 1 number
2018-11-08T11:52:27.4769290Z Passed app-funcs: add() Does the impossible
2018-11-08T11:52:27.5069526Z Results File: D:\a\1\s\TestResults\VssAdministrator_factoryvm-az489_2018-11-08_11_52_27.trx
2018-11-08T11:52:27.5094303Z
2018-11-08T11:52:27.5097933Z Total tests: 4. Passed: 4. Failed: 0. Skipped: 0.
2018-11-08T11:52:27.5098548Z Test Run Failed.
2018-11-08T11:52:27.5107869Z Test execution time: 6.5793 Seconds
2018-11-08T11:52:27.7938298Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2018-11-08T11:52:27.8666710Z ##[error]Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 1
2018-11-08T11:52:27.9914771Z ##[error]VsTest task failed.
2018-11-08T11:52:28.0305446Z Publishing test results to test run '213'
2018-11-08T11:52:28.0305575Z Test results remaining: 4. Test run id: 213
2018-11-08T11:52:28.0578818Z ##[section]Async Command Start: Publish test results
2018-11-08T11:52:28.2630565Z Published Test Run : https://linneyit.visualstudio.com/T-Level%20Setup%20Test/_TestManagement/Runs#runId=213&_a=runCharts
2018-11-08T11:52:28.2630806Z ##[section]Async Command End: Publish test results
2018-11-08T11:52:28.2631502Z ##[section]Finishing: VsTest - Front End
I wonder whether it is due to their being two test selector blocks (2018-11-08T11:52:18.4136511Z) and (2018-11-08T11:52:18.4347193Z), the first with Test filter criteria : null, the second with Test assemblies : **\chutzpah.json.
The task is setup as follows:
Visual Studio Test
Version: 2.*
Display name: VsTest - Front End
Select tests using: Test assemblies
Test files: **\chutzpah.json
Search folder: $(System.DefaultWorkingDirectory)
Test filter criteria:
[ ] Run only impacted tests
[ ] Test mix contains UI tests
Execution Options
Select test platform using: [x] Version [ ] Specific location
Test platform version: Latest
Settings file:
Override test run parameters:
Path to custom test adapters:
[ ] Run tests in parallel on multi-core machines
[ ] Run tests in isolation
[ ] Code coverage enabled
Other console options:
[x] Collect advanced diagnostics in case of catastrophic failures
Collect process dump and attach to test run report: On abort only
[ ] Rerun failed tests
Advanced execution options
Batch tests: Based on number of tests and agents
Batch options: [x] Automatically determine the batch size [ ] Specify a batch size
[ ] Do not distribute tests and replicate instead when multiple agents are used in the job
Reporting options
Test run title:
Build platform:
Build configuration:
[x] Upload test attachments
Control Options
[x] Enabled
[ ] Continue on error
Timeout: 0
Run this task: Only when all previous tasks have succeeded
Output Variables
Reference name:
Is the setup in this build pipeline task correct? Any other issues/possibilities as to why the task fails?
