Azure DevOps Pipeline: Tests that pass locally fail on the pipeline

Question

Our team is trying to implement Azure DevOps pipeline but we are having some issues. For background, we are running the tests on our self-hosted windows server. We got the front-end tests to all work but now we are having issues getting the backend to pass.

While the majority of the tests are passing just fine we have a few instances where it is not working as expected. Although they should be able to run in parallel, I turned that off to verify that wasn't the issue. I've also tried running them in isolation without any noticeable change.

The errors we get are just simple assert failures like the one below that don't give us much information.

 Error Message:
   Assert.AreEqual failed. Expected:<True>. Actual:<False>.

But the line numbers point to a mock setup like the one below which I don't understand.

mockQueryHandler.Setup(x => x.Handle(It.IsAny<FindQuery>())).Returns(info);

Currently the test class that is giving us the most problems is one that tests our event alert functionality. It seems that the first test will pass but the subsequent ones fail. However, we have the pipeline setup to rerun any failed tests three times until it gives up. So on every rerun the next test will pass. The only thing I can think of is that our auto mocks are giving us trouble for some reason. We set them up in accordance with AutoMock - How to unit test with Keyed Registrations?

We mock it out as follows:

  var IAbstractEventFactoryMock = new Mock<IAbstractEventFactory>();
  using (var mock = AutoMock.GetLoose(builder => builder.RegisterInstance(IAbstractEventFactoryMock.Object).Keyed<IAbstractEventFactory>("EventLogFactory").Keyed<IAbstractEventFactory>("ArpEventLogFactory"))) {
...
}

Below is our .runsettings file

<RunSettings>  
  <RunConfiguration>
    <ResultsDirectory>.\TestResults</ResultsDirectory>
  </RunConfiguration>

  <MSTest>
  </MSTest>
</RunSettings>

Below is the YAML we are using to run the tests

steps:
- task: VSTest@2
  displayName: 'Run All Tests'
  inputs:
    testAssemblyVer2: |
     **\*ID_Test*.dll
     !**\*TestAdapter.dll
     !**\obj\**
    searchFolder: '$(System.DefaultWorkingDirectory)/ID_Test'
    runSettingsFile: 'ID_Test/.runsettings'
    runInParallel: false
    runTestsInIsolation: true
    codeCoverageEnabled: false
    testRunTitle: 'Unit Tests'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    diagnosticsEnabled: true
    rerunFailedTests: true
  continueOnError: true

Again, these tests all pass without problems locally and they pass eventually if I run them enough times on the pipeline. So I don't think the problem is in the code itself. It seems to be that the tests aren't getting setup or cleaned properly. I am primarily looking for any ideas on other ways I can configure my yaml or runsettings to try to fix this issue. Thank you in advance for any help and let me know if I can provide any additional information to help.

Do you mean you are using self-hosted build agent and the tests pass without issue when you run locally? Do you run the tests on the self-hosted build agent machine? Could you try to clean the \_work folder of the build agent and try again? Also, try to configure a new agent to see how's the result. — Cece Dong - MSFT
Yes, when I run the test locally on my computer using Visual Studio's test runner they all pass. And we are using a self-hosted Windows build agent which is our server computer. That is a different computer than my own though, so I haven't been able to manually run the local tests on the server. I will look into testing that as well as the other solutions you mentioned, thanks for the ideas. — Andrew
Sorry for the delay, I was able to clean the _work folder but that didn't seem to have any effect. I also looked into running the tests locally on the server computer but my management said I can't have direct access to it. I will look into downloading an agent on my own computer to test that. — Andrew
I did find out that the server doesn't actually have Visual Studio installed. Could that create any problems? I feel like that shouldn't be an issue though because all the other tests work fine. — Andrew

Andrew Andrew · Accepted Answer · 2020-10-30T18:15:13

Turned out this was just an issue with our tests. In a few places we would use the current time which was fine locally, but the server had a much higher execution time that created a few errors. Testing the pipeline on an agent that had a similar execution time as running the tests locally helped me discover that it wasn't an issue with the pipeline.

Azure DevOps Pipeline: Tests that pass locally fail on the pipeline

2 Answers