Azure "KuduSync.NET" error using git deployment in multiple azure websites

Question

I have 4 azure websites each one with its own "staging" deployment slot. All of this staging slots are deployed using the same source code repository (git).

It's the same solution and each azure website compiles it's own project using the "Project" setting (azure website --> configuration tab --> application settings).

In fact almost always works well, but sometimes (10% of time) it throws me an annoying implementation error:

KuduSync.NET from: 'D:\local\Temp\a8b0a9fb-c31c-45a1-92d3-55b80315a6dc' to: 'D:\home\site\wwwroot' Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Web.config" Copying file: 'Web.config' Failed exitCode=1, command="kudusync" -v 50 -f "D:\local\Temp\a8b0a9fb-c31c-45a1-92d3-55b80315a6dc" -t "D:\home\site\wwwroot" -n "D:\home\site\deployments\6c8e6d6ba36c35d7adf1c0a2f93f5397df2600d1\manifest" -p "D:\home\site\deployments\52aaaa055bb165ee2b0b7ecbb78011b252ed001a\manifest" -i ".git;.hg;.deployment;deploy.cmd" An error has occurred during web site deployment.

I understand what it means, but it have no sense for me. The staging websites are almost never used (only for test purpuses) and the "Always available" setting is set to false.

For more details, some of the staging websites throws the error and some others not.

What I do for now is wait a little (5 minutes, 10, 15...), retry the deploy and pray to the god of the bits.

Anyone knows what's the problem?

Thanks in advance.

EDIT

I've modified the deploy.cmd following the steps proposed by @beatcracker and added two lines before the KuduSync step:

del "%DEPLOYMENT_TARGET%\web.config.bak"
rename "%DEPLOYMENT_TARGET%\web.config" "%DEPLOYMENT_TARGET%\web.config.bak".

It still doesn't work. This is the error message:

The process cannot access the file because it is being used by another process.

So it's not a kudu problem. I really don't know why is my web.config locked. I've tried to rename the web.config using the Debug console (*.scm.azurewebsites.net/DebugConsole) and the same error message is repeated.

If I kill the w3wp.exe process the deployment success.

Is there any way of knowing why is my web.config locked (¡¡sometimes!!)?

** UPDATE 2 ** I thought that @beatcracker was the correct one but I'm still struggling with the problem.

I've added this two lines to my deploy.cmd

cd %DEPLOYMENT_TARGET%
for /F "tokens=3,6 delims=: " %%I IN ('%DEPLOYMENT_SOURCE%\Handle.exe -accepteula Web.config') DO %DEPLOYMENT_SOURCE%\Handle.exe -accepteula -c %%J -y -p %%I

Note that I needed to add the -accepteula argument on the DO statement and I needed to do "cd %DEPLOYMENT_TARGET%" because it didn't work if I set it in the IN statement.

And the new problem is...

Error closing handle: A

I've been Googling and I didn't find any description of an "A" error.

Any idea?

Answer 1

Are you using the latest version of KuduSync.Net? Because this commit: "Try to move a file to a known directory if it is locked" is supposed to fix the issue (Files locked during deploy from GitHub).

Update #1:

I'm using Azure, so I suppose that they use the last version of KuduSync

This could be, or could be not. Because you don't have direct access to KuduSync, you need some workaround. Here is idea: create your custom deployment script, that will rename web.config before update to web.config.bak (and delete previous web.config.bak if any).

1. Get your deployment script (deploy.cmd) from Kudu: Custom Deployment to Azure Websites

   1. Access the site's Kudu Services at: yourwebsite.scm.azurewebsites.net
   2. Click Tools > Download Deployment Script and get the deployment script Azure generated for your code.

2. Modify it to rename web.config before updating files

3. Commit it to the root of your repo. Now Kudu will use it instead of generic one.

Here is good roundup of custom deployment with Azure (with step by step instructions in part 3): 1, 2, 3

Update #2:

I've modified the deploy.cmd following the steps proposed by @beatcracker and added two lines before the KuduSync step ... It still doesn't work.

Sigh. The IIS monitors files\folders to intercept changes and restart AppDomains\AppPools accordingly. The monitoring logic is complicated, to say the least. So you have two ways for dealing with this:

1. You can spend undefined amount of time debugging exactly why this lock happens. The link above has ASPX page code sample that will display the number of requests, number of AppDomain restarts, and number of active directory monitors. And here is a brackets issue thread on GitHub, where similar behaviour is discussed. It may provide some insights.

2. You can apply quick and dirty workaround, and move on to more important things in your life. I can think of at least two:

   1. Recycle application pool in your deployment script: appcmd recycle apppool YourAppPool.

   2. Deploy handle.exe to your servers and use it in custom deployment script to unlock the file:

      for /F "tokens=3,6 delims=: " %%I IN ('handle.exe -accepteula %DEPLOYMENT_TARGET%\web.config') DO handle.exe -c %%J -y -p %%I
      

   To simplify deployment, you can include handle.exe in your git repo, so it would be automatically deployed.

Relevant XKCD: