Azure DevOps PublishTestResults task -- how to publish results if tests fail

Question

I am running a pytest-based suite of tests during my Azure DevOps build process. I have two jobs arranged to run these tests against two different environments.

In each job, I run the pytest tests using a script task and generate a junit-style xml output file, then have a PublishTestResults task publish that xml file. This is working great, and I'm able to peruse my test results in the azure build tests report UI -- but only if all the tests pass. If any tests fail, the publish task is skipped, and the tests aren't reported in the UI.

YML extract:

- job: 'RunTestsQA'
    continueOnError: True
    steps:
    - task: UsePythonVersion@0
      inputs:
        versionSpec: '3.6'
        architecture: 'x64'
    - task: DownloadSecureFile@1
      inputs:
        secureFile:  'ConfigFile'
    - script: pip install -r requirements.txt
      displayName: 'Install Requirements'
    - script: |
        pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
      displayName: 'Test with pytest'
    # PUBLISH JUNIT RESULTS
    - task: PublishTestResults@2
      inputs:
        condition: succeededOrFailed()
        testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
        testResultsFiles: '**/TEST-*.xml' 
        #searchFolder: '$(System.DefaultWorkingDirectory)' # Optional
        mergeTestResults: false # Optional
        testRunTitle: 'API_CHECK QA'
        #buildPlatform: # Optional
        #buildConfiguration: # Optional
        publishRunAttachments: true # Optional

Through some experimentation, I've been able to confirm the XML file is always created. What do I need to fix here? A test report isn't super helpful if it only shows up when the tests pass.

Answer 1

In your task description, the condition is effectively listed as a task input, and hence won't be taken into account at all.

You had:

# PUBLISH JUNIT RESULTS
- task: PublishTestResults@2
  inputs:
    condition: succeededOrFailed()
    testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
    testResultsFiles: '**/TEST-*.xml' 

The correct setup is

# PUBLISH JUNIT RESULTS
- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
    testResultsFiles: '**/TEST-*.xml' 
  condition: succeededOrFailed()

The full list of things you can do with conditions is here

Answer 2

I'm using Ruby and Minitest, but I have found that the following setting allows the PublishTestResults task to run:

- script: |
    pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
  displayName: 'Test with pytest'
  continueOnError: true

The only issue that I have found with this setting is that if the build fails, it reports as "Partially Succeeded" and not "Failed".

edit:

Of course, if your build process has any deploy tasks after the test task, you may not want to use this setting.