TFS 2012, TFSBuild.proj, UpgradeTemplate.xaml process definition and Coded UI Tests

Question

Another few TFS questions if anyone can help even remotely :-)

We currently have TFS 2012 installed. We have 2 servers (both VMs running Windows Server 2008 R2 x64). One server is TFS with the application layer and database (running SQL Server 2008 R2) installed. The other server is our BUILD server with the build controller and build agent installed on it.

We can connect to TFS 2012 OK from our local developer PCs using Visual Studio 2012 Ultimate.

We then have various complex build definitions that compiles our code, runs unit tests, generates real-time sandcastle documentation and installs to our multi tiered, multi server TEST or LIVE environments using InstallShield (takes SSL and performs pre-install conditions etc)

We are using the UpgradeTemplate.xaml build process definition along with TFSBuild.proj file. FYI it was originally developed in TFS 2008 and has served us well hence why we don't WANT to upgrade to a modified defaulttemplate.xaml

We now want to include "post deployment tests" (aka smoking testing or system testing or end-to-end testing I believe) in the form of Coded UI tests.

1. We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?

2. I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an internative process and not a service). Is this correct?

3. If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?

4. Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?

Answer 1

1. We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?

If the coded ui test is used as a stand alone unittest in the unittest dll, mstest should pick up the coded ui tests. If the coded ui test is however an automation of a test case, an alternative is to use the TCM command line tooling to kick of the test case and thereby executing the coded ui test.

2. I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an interactive process and not a service). Is this correct?

Yes: You would need an interactive build agent. No: the current build agent could also be set to interactive.

3. If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?

Yes: It works as the articles imply No: The intellitrace collectors and logs are usually more useful than a video recording. The articles also forget to mention that your TFS server might grow out of proportions if not handled with care. http://geekswithblogs.net/terje/archive/2011/11/15/guide-to-reduce-tfs-database-growth-using-the-test-attachment.aspx

4. Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?

No, there not the same and don't try to pretend that (like blend). The focus of the tools are different; MTM is focused on testing by a tester (functional, manual, exploratory), testing in visual studio is focused on testing as a developer (unittest, webtesting, automating tests). Visual Studio ultimate therefore can't edit and create test cases and suites. MTM can't create develop (coded) unittest tests. Installing visual studio ultimate also installs MTM, so you can use both if needed.