Any mechanism for Run Functional Test Task to not fail the release deploy on test failure

Question

Under the XAML and earlier builds we had built into the default templates a flag to state if failed tests failed the build, or issued a warning.

Under Release Management 2015 in a release definition using the Run Functional Test task- I don't see a means to indicate that failed tests should not fail the release. Am I overlooking something? A flag, an argument, is it something I don't realize I can set in a test settings file?

The flag of "Continue on error" for the task will still cause a failure of the release deploy for that environment if the task fails which it does if a test fails. Unless I am wrong on this but I think I am correct as I tested that scenario. TFS 2015 Update 3.

My scenario which I suspect is not uncommon is I run a few different sets of test on a deploy to a new environment. For example: set 1 are smoke BVT and if they fail I want the deploy to fail. That works fine. But set 2 I add a separate task to run that set of Functional Tests but they are regression tests that I don't want to fail the deploy just because one of them fails.

Worst case scenario I'll modify the task and make a custom task with the logic I desire but I'd prefer to not have to keep maintaining that if there is a means I am missing of flagging that the task failure does not fail the deploy.

Answer 1

Build and release tasks have an option to Continue on error in the Control Options parameters for each task. In a build definition, this results in a Build partially succeeded result if a task with this option set should fail. The same behavior is also available in release definitions. If a task fails, the overall release result will show as "Release partially succeeded".

By default, a partially successful release will not automatically trigger a release to a subsequent environment, even if this behavior is specified in the environment deployment options. However, a new option can be set in each release environment that instructs Release Management to trigger a release to a subsequent environment when the previous release is partially successful.

For more details, see Trigger on partially successful deployments

It's the simplest way in your situation. Otherwise you may have to modify the task and make a custom task with the logic as you mentioned. However this feature is not available on TFS2015 update3, you may have to update to TFS2017. Please see the related Release Note.