How can we read the Azure DevOps task log?

Question

I have a task in an azure pipeline that conducts some dynamic tests but since the tests are multiple in number, whether the individual test passes or fails, that is determined by whats there in the logs.

As such the Azure DevOps steps always turns green. I was looking to solve a problem by reading that log and then deciding what to do next based on whether the logs contain failure or not. So let's say, if the task turns green and logs detect failure, I will want to take a decision to not publish an artifact.

How can I solve this problem in a standard way by reading the logs of a previous task using a standard azure api in the next task and then taking a decision?

Clarification: as needed

These tests are not units, they are actually some custom java tests which are triggered via calling a .sh file from a self-hosted agent on a linux machine.

Answer 1

If you do not have a way to fail the pipeline with the standard error, you can use the Builds - Get Build Log to investigate the task and according to the results set a variable, in the other tasks, uses custom conditions with this variable, or just fail all the pipeline.

So, a PowerShell script with something like that:

$token = "YOUR-PAT"
Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))
 
# Construct the REST URL to obtain Build ID
$uri = "$(System.TeamFoundationCollectionUri)/$(System.TeamProject)/_apis/build/builds/logs/{logId}?api-version=5.1"
 
# Invoke the REST call and capture the results
$log= Invoke-RestMethod -Uri $uri -Method Get -ContentType "application/json" -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}

if($log -match "some error")
{
    # Option 1: set a variable
    Write-Host "##vso[task.setvariable variable=testsStatus]failed"
    # Now in the artifacts task use custom condition:
    # and(succeeded(), ne(variables['testsStatus'], 'failed'))

    # Option 1: fail the pipeline
    Write-Error "The tests not passed!"
}

(you can get the log id with the API Builds - Get Build Logs)

Answer 2

Edit: after clarification, I split my comment into two parts: your custom test case, and then the rest of my comment which was previously here, applicable for cases where standard unit testing frameworks are used.

Alternative 1: Solution possibility for custom testing, based on analyzing test log files

First you will need a script, e.g. PowerShell script, which interprets the log files, determines if tests have failed or not, and sets vso task.complete results property accordingly (see example below).

Let's call this file evaluate-test-results.ps1, and let's assume it interprets some sort of testResults.json data, containing an important property named haveTestsPassed, deciding whether or not the tests passed:

evaluate-test-results.ps1 file content:

<#
.SYNOPSIS
Checks test results.
#>

param (
    [Parameter(Mandatory = $true)]
    [string]$testResultsJson
)

try {
    $testResults = $testResultsJson | ConvertFrom-Json

    if($testResults.haveTestsPassed)
    {
       Write-Host "##vso[task.complete result=Succeeded;]Tests have successfully passed"
    }
    else
    {
       Write-Host "##vso[task.complete result=Failed;]Tests have NOT passed"
    }
}
catch {
    Write-Host "##vso[task.complete result=Failed;]Error while parsing test results from JSON log file..."
}

Then, you can use this script in a PowerShell@2 task, like this:

- task: PowerShell@2
  displayName: "PowerShell: Evaluate test results from testResults.json"
  inputs:
    targetType: filePath
    filePath: 'evaluate-test-results.ps1'
    arguments: "-testResultsJson 'testResults.json'"
    failOnStderr: true

Note:

You can also throw standard error when haveTestsPassed is false, alternatively, thanks to the failOnStderr property. However, this solution is a bit more official above.

Alternative 2: Solution possibilities for standard unit test frameworks

Simpler solution if dotnet test command is configured to run your tests

The simple solution would be like this, if your tests are bound to the dotnet test command, using for example xUnit. In such a scenario, this task will fail by default if one of your unit tests fail, without ever needing PublishTestResults@2 task:

# This task will run the 'dotnet test' command, as if you would from a CLI
- task: DotNetCoreCLI@2
  displayName: Run and publish tests
  inputs:
    command: 'test'

If you can't use dotnet test command

In this case, you have to use PublishTestResults@2 task to interpret test results. Set your testing framework/tool and test results file accordingly. Be sure to set failTaskOnFailedTests: true

- task: PublishTestResults@2
  inputs:
    testRunner: VSTest # or else...
    testResultsFiles: '**/*.trx' # or else...
    failTaskOnFailedTests: true