MSBuild & TeamBuild - BuildInParallel failing because of MSB3021 file permission violation

Question

I maintain the build of a fairly large piece of software, consisting of roughly 350 csharp projects. Our build time for a debug built clocks in at about 17 minutes.

I have been looking at ways to improve build time, and the BuildInParallel property did look intriguing. Especially since we have a quad-core server doing our builds, it should really be able to take advantage of the computing power.

But alas... After setting the property, modifying the config file for the build agent and restarting it, the first run really did look promising, way faster than normal, right up until the point where it failed.

After looking at the build logs, it looks like the build fails when it attempts to copy references marked as CopyLocal=true to the ouput dir. If C# project A and C# project B are built in parallel, and the both reference the same third-party dll, and tries to copy it at the same time, the second process to attempt to copy the file will get a file access violation - the file is being used by another process.

Anyone experienced this, and been able to get multi-proc builds working on Team Build?

---

Here's one of the failures, kind of hard to figure out which other project was being built at the same time.

I have removed all the non-relevant stuff:

54>Target "_CopyFilesMarkedCopyLocal" in file "c:\WINDOWS\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets" from project "d:\temp\PCM\1.3-Maint_CI\Sources\Modules\Core\Test\UnitTest\TestDIPS.Core.Data.Server.NUnit\TestDIPS.Core.Data.Server.NUnit.csproj": 54>Task "Copy" Copying file from "..........\Bin\3rdParty\Oracle\Oracle.DataAccess.dll" to "d:\temp\PCM\1.3-Maint_CI\Binaries\Debug\Oracle.DataAccess.dll". Command: copy /y "..........\Bin\3rdParty\Oracle\Oracle.DataAccess.dll" "d:\temp\PCM\1.3-Maint_CI\Binaries\Debug\Oracle.DataAccess.dll" 54>c:\WINDOWS\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets(2703,9): error MSB3021: Unable to copy file "..........\Bin\3rdParty\Oracle\Oracle.DataAccess.dll" to "d:\temp\PCM\1.3-Maint_CI\Binaries\Debug\Oracle.DataAccess.dll". The process cannot access the file 'd:\temp\PCM\1.3-Maint_CI\Binaries\Debug\Oracle.DataAccess.dll' because it is being used by another process. Done executing task "Copy" -- FAILED. 54>Done building target "_CopyFilesMarkedCopyLocal" in project "TestDIPS.Core.Data.Server.NUnit.csproj" -- FAILED.

Answer 1

The default targets that ship with MSBuild are designed for CopyLocal behavior - which is what VS relies on. CopyLocal is problematic when you're outputting to a single output directory.

To be able to truly build in parallel, you'll need to disable several CopyLocal specific behaviors in the Microsoft.*.Common.*.targets files. I've spoken with some folks in the MSBuild team in the past, and this is a particularly hairy thing to do. Even if you do blank-out some of the CopyLocal behavior, the VS test accessors don't behave well with build in parallel.

Some of the things you can start with:

1. Disable CopyLocal behavior for references by setting private to true.
2. Disable CopyLocal for CopyToOutputPath files when building dependent Projects.

Answer 2

MS Build Copy task has undocumented feature, at least Google keeps silence. If set system wide environment variable MSBUILDALWAYSRETRY=1 This task will retry to copy a file even it gets Access Denied exception during copy operation

Example of output

C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Common.targets (3513): Got System.UnauthorizedAccessException: Access to the path 'C:\Builds\8\28\Binaries\Release\fr\System.Spatial.resources.dll' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
   at System.IO.File.Copy(String sourceFileName, String destFileName, Boolean overwrite)
   at Microsoft.Build.Tasks.Copy.CopyFileWithLogging(FileState sourceFileState, FileState destinationFileState)
   at Microsoft.Build.Tasks.Copy.DoCopyWithRetries(FileState sourceFileState, FileState destinationFileState, CopyFileWithState copyFile) copying C:\Builds\8\28\Sources\Main\Solutions\packages\System.Spatial.5.2.0\lib
et40\fr\System.Spatial.resources.dll to C:\Builds\8\28\Binaries\Release\fr\System.Spatial.resources.dll and HR is -2147024891
 C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Common.targets (3513): Retrying on ERROR_ACCESS_DENIED because MSBUILDALWAYSRETRY = 1
 C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Common.targets (3513): Could not copy "C:\Builds\8\28\Sources\Main\Solutions\packages\System.Spatial.5.2.0\lib
et40\fr\System.Spatial.resources.dll" to "C:\Builds\8\28\Binaries\Release\fr\System.Spatial.resources.dll". Beginning retry 1 in 1000ms. Access to the path 'C:\Builds\8\28\Binaries\Release\fr\System.Spatial.resources.dll' is denied.

Answer 3

I have the same problem many months ago.

There are two types of file copy in VS C# project files (.csproj files):

1. Referenced Assemblies which has a "Copy Local" property. when the property is True, the assembly is copied into output path.

2. Additional files which has a "Copy to Output Directory" property. when the property is set to "copy always" or "copy if newer" the file is copied into output path.

problem:

A: If two or more projects are built concurrently, and two of them tries to copy the same file into output directory, you may faced with the errors like "error MSB3021: Unable to copy file", "Access to the path is denied", "The process cannot access the file", etc.

B: If two or more projects refrences a common project which has some items of type 2. In this case during the parallel build of the projects, two of them may tries to build the "GetCopyToOutputDirectoryItems" target of the common project concurrently. So you may again faced with the above exceptions.

Solution1:

force the OutputPath of different .csproj files not to be in tha same path

http://social.msdn.microsoft.com/Forums/is/tfsbuild/thread/a62a6f98-ec44-46c1-a0d0-7f441f0db973

solution2:

Step1: The solution is to set the property of common Items (to False in case 1 and to "dont copy" in case 2) for all but one of these common items in your projects. or remove them If it is possible.

In order to find potential mistakes, you can search the words "private" (for case 1) and "CopyToOutputDirectory" (for case 2) in files *.csproj

Step2: ...

Good Luck

Answer 4

It looks like that your projects are configured to the same output directory. If you configure Project A and Project B to output to different directories, this would resolve this error.

Answer 5

This can also occur due to the file being Read Only. In those cases (i.e., not the ones covered by the bulk of the other answers), there are appropriate workarounds which vary depending on whether you're able to rely on MSBuild 3.5 or later.

Answer 6

Which compiler are you using? As far as I know, up until VS2005 parallel build doesn't work for multiple projects. It's a known issue that MS keeps saying they'll fix eventually, but I have no idea whether or not they actually solved it in VS2008.