Adding App.Config to VS 2015 CodedUI Project breaks Test Execution on Test Agent 2013

Question

I have the below setup

Client - Visual Studio 2015 Or MSTest.exe (14.0)

Test Controller - VS 2013 TC

Test Agent - VS 2013 TA (Remote)

Things work fine when I try to execute tests locally & remotely as long as the test project doesn't have 'App.Config'. (Regardless of content of App.Config)

Problem shows up When I try to execute tests 'Remotely' (By using a testsettings file with Test Controller & Test Agent).

Whenever the test project has an 'App.Config' file included, the remote test execution fails with 'dll not found exceptions' (local execution is fine with App.Config)

for e.g.

Unit Test Adapter threw exception: Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified..

App.Config (On the sample CodedUIProject)

<?xml version="1.0" encoding="utf-8" ?> <configuration> </configuration>

These issues are expected to be resolved (through Assembly Binding entries in exe configs of Agents and Controller with updates 5 ) however I am out of luck.

I have tried adding these DLLs to the deployment (via TestSettings file) which will throw exception for another 14.0 version DLL.

Any insight would be helpful on why codedUIProject.dll.config would interfere with Assembly Binding? or if I am missing something trivial ?

Answer 1

If anyone cares, I got it working by

1. Adding Assembly Rebinding from 14.0 to 12.0 in the App.Config for CodedUI DLLs mentioned here and
2. Xcopying the PublicAssemblies of VS 12.0 (C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies) to the your probing location mentioned in Step 1

This seems to be a bug with Microsoft VS 2015 compatibility with 2013 controllers and 2013 agents.

Answer 2

Good job solving the issue for yourself - the 2013 and 2015 gap in Visual Studio/Test Agents/Test Controllers is inexcusably bad. It's not necessarily a "bug" as much as it is a horrible design decision on Microsoft's part.

One other solution I've tried to inform people of multiple times for CodedUI is creating your own NuGet Package (or equivalent solution) to refer to from your solution. This makes them as portable as possible.