TeamCity + artefacts + chained builds + MSBuild + csproj/dll referencing issue

Question

I am writing a build/ci/deploy process for very large system (300+ projects). This is a C# .net project and we are using Visual Studio for development IDE and MSBuild (from psake tasks) to compile our solutions.

There are multiple solutions containing multiple projects, to make matters worse some of projects are referenced as .csproj files and some are referenced as .dll. First step of build refactoring will be achiving that all projects will reference each other through the csproj file. This way I could create one master solution containing all the projects and use parallel build supported by MSBuild, this would serve perfect for CI build.

But for deploy builds I want to use chained builds and artefact dependencies.

I have imagined my build like:
1. Build CommonLibrarry
1.1 Store .dlls in artefacts as common.zip
2. Use artefacts from 1.1 and build CoreProduct
2.1 Store .dlls from CoreProduct in artefacts as product.zip
...etc (20 more steps like these)...

The problem is that CoreProduct has CommonLibrary references as .csproj and those .csproj files will not be present as CoreProduct build, and why would it be since the CommonLib has already been built and artefacts are ready. The MSBuild project file (CSPROJ file) contains information about files that needs to be compiled and about references projects not the solution file, therefore I cannot create separate .csproj file for deployment system, developers would use and add their class files to different project file and I would have to administrate the difficult sync between those two project files (e.g. real.csproj and real.CI.csproj). I could use T4 templating for generating both files but this way I would loose easy file adding from the Visual Studio.

Conditional sections in .csproj is out of the picture because there is over 50 developers and learning them that adding a reference now requires tweaking .csproj file is impossible :)

I’m searching for the patters/ideas how to solve this issue

Answer 1

As you figured out already yourself, both requirements (only project references + chained builds) contradict each other. At least I can't think of a way to get them to play togeher. So one of them will have to change somehow.

First idea: do you really need chained builds? It takes a while to set up, in the end you probably want to build everything anyway, so why not just build everything by default?

If the answer to the above is 'yes': suppose not using project references after all. Using dll references directly is a mess though, wouldn't recommend it. You could use NuGet in some way though. That would mean for instance CommonLibrary is built into a package, and other builds use that package through NuGet meaning it's still a dll reference but install is automatic. I've seen people do that with TeamCity (since it can act as package server) but I don't know the details of the setup, not how to deal with local dev builds. It does mean teaching your dev team to add references only using NuGet of course.

Another way: you keep project references (yay!), but for chained builds you run a script on all project files that removes the reference and replaces it with an assembly reference. Sound messy, but I'm usig something like this as well and once setup it's pretty much foolproof. In practice you'd add some build steps to TC so you get a build flow like this:

• checkout CommonLibrary, build and store artefacts
• checkout CoreProduct, run script, build and store artefacts
• checkout XYZ, run script, build and store artefacts

And the script would:

• look for ProjectReference items
• store the value of the Name tag
• remove the ProjectReference items
• for every Name found above, add a plain reference like <Reference Include=name HintPath=/path/to/artefacts>
• save the project file

Writing such a program is relatively easy if you use C# and the classes from the Microsoft.Build.Construction namespace since they can deal with project files. Or else you can use your favourite xml library.

edit

in response to your comment/answer: it seems your build environment is a bit crippled. You can either spend lots of time figuring out workarounds etc, or you could just fix it properly once and for all. From my experience the first option leads to more and more problems and in the end you're left with a monolithic block where even the slightest change results in a waterfall of problems requiring again more time to fix everything. The second option on the other hand (eg do not rely on SolutionDir and place targets in a common place accessible by all projects - don't have different projects output assemblies with the same name, ...) might take more time now, but it will be worth it in the end casue you'll be left with a fairly clean build system that is easy to extend and you won't be wasting more time in the future. Also if you fix said problems now, chances are you can simply use something like my last solution and you do not need Choose element/multiple projects/conditions/... etc: the script creates a whole new project file that is just used for the chained build and nothing else.

Answer 2

@stijn:

Since this message is too long for comment I'm writing it as an answer.
Thank you for responding and all your suggestions. As you mentioned using NuGet would complicate developer's environment, so this is out of the picture. I'm using Microsoft.Build library switching references from .dll to .csproj to build one "master" solution. I like your idea and I've been thinking about it by myself too, the problem occurs when importing targets and they are dependent on the MSBuild's SolutionDir property, there is inconsistency with naming project and we have two different projects(.net version) and same AssemblyName, ....
I could go on with the conditional builds and recreate new solution files, e.g. *.TeamCity.sln, *.Dev.sln
I could use conditions with imports, references, ... I could create a tool that will automatically remove new references and add them into Choose section, deleting reference from the VisualStudio is done property (it deletes ProjectReference tag in the Choose section), and my after commit tool could detect those changes. Since it is the company's policy that all projects have prefix I can easily distinguish which projects are relevant to me. Fairly big problem is also that Microsoft.Build.Evaluation does not support Choose sections so I would have to write the whole project parsing with e.g. XDocument.

I'm still not 100% satisfied with this solution and I'm still open for suggestions.