How to executes NUnit tests in parallel with one agent in Azure Devops

Question

We have 2 local computers that are used as agent. Both computers have one agent. Those computers have at least 12 cores and we would like to run our tests in parallel, only on one computer. So my understanding is that there is no additionals agents needed, since I can execute Nunit directly in a parallel fashion.

In the documentation I found: https://docs.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops The notion of running tests in parallels and on multiple agents seems linked. I don't want to use multiple agents as it prevents using isolation.

Is there a way to avoid it? Use more than one core of one agent?

My current yml description:

  - task: VSTest@2
    timeoutInMinutes: 300
    inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: '*.Test*.dll'
      searchFolder: '$(System.DefaultWorkingDirectory)/$(buildConfiguration)'
      codeCoverageEnabled: false
      platform: 'Any CPU'
      uiTests: false
      configuration: '$(buildConfiguration)'
      rerunFailedTests: false
      pathtoCustomTestAdapters: 'Solution/packages/NUnit3TestAdapter.3.12.0/build/net35'
      minimumExpectedTests: 1000
      runInParallel: true
      runTestsInIsolation: true
      failOnMinTestsNotRun: true
      resultsFolder: 'testResults'
      runSettingsFile: '$(Build.SourcesDirectory)\azure-tests.runsettings'

Answer 1

If you want to enable parallelism across agents you need to state that explicit on job configuration:

jobs:
- job: ParallelTesting
  strategy:
    parallel: 2

If you didn't do that your tests run on one agent, which you want if I understood you.

Answer 2

Setting aside parallelism across agents, NUnit itself has two kinds of parallelism: between separate processes and within one process.

The first kind is unfortunately what NUnit also calls "agents" but has nothing to do with the agents your question is talking about. In any case, that's not available to you when running tests through the NUnit 3 VS adapter because it relies on the host to handle processes rather than creating one itself.

This leaves the second kind of NUnit parallelism: use of multiple threads within the test process. This is mainly controlled by your use of attributes in the test assembly.

[Parallelizable] tells NUnit that it is safe to run a particular test in parallel with others. If you are not using it already, it takes a fair bit of work to determine which tests may run in parallel within the same process.

You can use LevelOfParallelismAttribute on the assembly to specify the number of parallel threads to be used or let it default to the number of cores available.

You'll have to determine for yourself whether it's worth the effort to parallelize your tests in this way as opposed to configuring multiple agents on each machine.

Another alternative would be to run using NUnit 3 Console rather than the vstest runner. In that case, each test assembly defaults to running in a separate process.