Edit: after clarification, I split my comment into two parts: your custom test case, and then the rest of my comment which was previously here, applicable for cases where standard unit testing frameworks are used.
Alternative 1: Solution possibility for custom testing, based on analyzing test log files
First you will need a script, e.g. PowerShell script, which interprets the log files, determines if tests have failed or not, and sets vso task.complete results property accordingly (see example below).
Let's call this file evaluate-test-results.ps1
, and let's assume it interprets some sort of testResults.json
data, containing an important property named haveTestsPassed
, deciding whether or not the tests passed:
evaluate-test-results.ps1 file content:
<#
.SYNOPSIS
Checks test results.
#>
param (
[Parameter(Mandatory = $true)]
[string]$testResultsJson
)
try {
$testResults = $testResultsJson | ConvertFrom-Json
if($testResults.haveTestsPassed)
{
Write-Host "##vso[task.complete result=Succeeded;]Tests have successfully passed"
}
else
{
Write-Host "##vso[task.complete result=Failed;]Tests have NOT passed"
}
}
catch {
Write-Host "##vso[task.complete result=Failed;]Error while parsing test results from JSON log file..."
}
Then, you can use this script in a PowerShell@2
task, like this:
- task: PowerShell@2
displayName: "PowerShell: Evaluate test results from testResults.json"
inputs:
targetType: filePath
filePath: 'evaluate-test-results.ps1'
arguments: "-testResultsJson 'testResults.json'"
failOnStderr: true
Note:
You can also throw standard error when haveTestsPassed is false, alternatively, thanks to the failOnStderr property. However, this solution is a bit more official above.
Alternative 2: Solution possibilities for standard unit test frameworks
Simpler solution if dotnet test
command is configured to run your tests
The simple solution would be like this, if your tests are bound to the dotnet test
command, using for example xUnit. In such a scenario, this task will fail by default if one of your unit tests fail, without ever needing PublishTestResults@2
task:
# This task will run the 'dotnet test' command, as if you would from a CLI
- task: DotNetCoreCLI@2
displayName: Run and publish tests
inputs:
command: 'test'
If you can't use dotnet test
command
In this case, you have to use PublishTestResults@2
task to interpret test results. Set your testing framework/tool and test results file accordingly. Be sure to set failTaskOnFailedTests: true
- task: PublishTestResults@2
inputs:
testRunner: VSTest # or else...
testResultsFiles: '**/*.trx' # or else...
failTaskOnFailedTests: true