Azure Devops 2019 Cannot Enable Diagnostics in Visual Studio Test Task

Question

We have an on-premises, TFVC setup with Azure Devops 2019. We use the "Classic Editor" to create our pipelines since YAML does not support the TFVC repository: https://developercommunity.visualstudio.com/t/enable-yaml-for-tfvc/234618

We have a sporadic crash during the test run that brings down the entire suite. We're just using MSTest and the Visual Studio Test Task as you can see below. As you can see, I have enabled the "Collect advanced diagnostics" option:

However, as you can see in the output, the option is only enabled to 'true' on the first pass of 'Run the tests locally' where no assemblies are found? On the second pass, when it finds the assemblies, the option is 'false' and the command line is not updated.

How can I get diagnostics enabled for the second pass? There's only one Visual Studio Test task in the pipeline and it seems it picks up the /InIsolation parameter fine so I was surprised to see the diagnostics option disabled. Why are there two passes happening anyway?

Thanks.

UPDATE 1

After turning on system.debug, I noticed a few dlls being found that contained the word "test" but were not test assemblies so I changed **\*test*.dll to **\*Tests.dll to clean this up. After this, I finally saw the following in the log file:

/diag:"H:\visbuild2\_work\_temp\f7742940-b794-11eb-9d69-699b53f653ca.txt"
Starting test execution, please wait...
Logging Vstest Diagnostics in file: H:\visbuild2\_work\_temp\f7742940-b794-11eb-9d69-699b53f653ca.txt
Logging TestHost Diagnostics in file: H:\visbuild2\_work\_temp\f7742940-b794-11eb-9d69-699b53f653ca.host.21-05-17_22-53-21_51202_1.txt

...

Results File: H:\visbuild2\_work\778\s\TestResults\tfsservice_VIS-BUILD_2021-05-17_22_53_25.trx

Total tests: Unknown. Passed: 353. Failed: 0. Skipped: 0.
Test Run Aborted.

So this is progress but the issue now is that the H:\visbuild2\_work\_temp location is above the current build location and seems to be shared among all builds so the files were quickly deleted by another build. I tried changing the location via the 'Other console options' like so:

/Blame:CollectDump;CollectAlways=true;DumpType=full /Diag:"$(BuildLocation)\testlog.txt";tracelevel=verbose

But this produced the error that /Diag cannot be specified twice. I then tried unchecking the "Collect advanced diagnostics" option but strangely, I got the same error that /Diag was specified twice. Is there a way to change the location where the diagnostics file is produced? Of course, I still have no indication that the crash dump was created despite the test run having been aborted.

UPDATE 2

It seems that /diag is only enabled with system.debug is set to true: https://github.com/microsoft/vstest-docs/blob/master/docs/diagnose.md

Answer 1

Try to set the value of the "Collect process dump and attach to test run report" field to "Always" to see if it can work.

Answer 2

After a bunch of trial and error, I finally got diagnostics enabled.

First, enabling system.debug will enable the /diag flag. The /diag flag is not enabled by checking "Collect advanced diagnostics" as I assumed. I have set system.debug to false.

Second, I redirected the diagnostic files to a custom location using the "Other console options" as follows:

/Blame:CollectDump;CollectAlways=true;DumpType=full /Diag:"$(BuildLocation)\testlog.txt";tracelevel=verbose

The /Blame option is useful as it produces an XML showing test execution order with the last test being the currently running test during the crash.

Third, I kept the "Collect advanced diagnostics" option checked.

Fourth, install procdump.exe (https://docs.microsoft.com/en-us/sysinternals/downloads/procdump) on the build server and added a PROCDUMP_PATH environment variable to point to the executable in order to produce the dump file for analysis.