Azure Web Site Kudu Publishing failing with timeout

2
votes

I have a custom deployment script (*.sh script) defined for my azure deployment.

Just today, I have found that I am unable to publish. I updated my bitbucket repository and after a while I get an error similar to the following:

Command 'starter.cmd deploy_pvl_cont ...' was aborted due to no output nor CPU activity for 180 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.\r\nstarter.cmd deploy_pvl_content.sh

I have tried a number of things to try to diagnose the problem.

  • Increase SCM_COMMAND_IDLE_TIMEOUT to 300
  • Run the script locally (Works)
  • Set up a new fresh deployment slot and try publishing same commit (Same error)
  • Tried publishing the previously successful commit (Same error)
  • Look for useful error messages in a diagnostic log dump (Coldn't find anything more useful)
  • Tried running the deployment script from the Kudu Console (No output returned, like it didn't actually run)
  • Tried reverting git to a previous version as suggested by @david-ebbo
  • Tried simplifying my script to a single echo command with the same results

Not sure what I can do to debug this further. Ideally I would like to get the output of the shell script on the azure host but don't know how to get it. Any ideas?

1
azurekudu
How long does it take to execute this script on local machine – Brij Raj Singh - MSFT
On my local machine it takes around a minute. Previous publishes on Azure have taken 180 seconds. Note: I just edited my question to show that I did try increasing the SCM_COMMAND_IDLE_TIMEOUT and while it did wait longer it failed with the same message. – Sudsy
If you're able to find the specific command that triggers this and repro it directly from Kudu Console outside of your script, that would be a good step to isolate the issue (e.g. see here). Is your script batch or bash? – David Ebbo
While it's stuck, you can also look at Kudu Process explorer which might indicate which process it's stuck in. Also, can you share your web app name, either directly or indirectly? This may help us investigate. Thanks! – David Ebbo
Hi David, the site is stage.pvlighthouse.com.au. I will look into the other suggestions soon. – Sudsy

1 Answers

0
votes

Updated answer

This is a regression caused by the move to git 2.8.x in Azure. The issue is tracked by https://github.com/projectkudu/kudu/issues/2041.

Here is a very simple workaround (and you don't need to bring in the old git tools): instead of setting your COMMAND to deploy_pvl_content.sh, set it to bash deploy_pvl_content.sh

We'll address the issue, but this workaround will get you going.

Original answer (only leaving for context)

You could be running into some flavor of this issue, which is caused by the upgrade to git 2.8.1 that we just did.

While we're trying to get to the bottom of it, please try this workaround to see if that helps:

  • Go to Kudu Console
  • Create a d:\home\bin folder
  • Copy the old Windows git 1.8.x folder in there. You can get the content from here. If you drag and drop the zip into Kudu console, there is a special unzip drop area that will expand it.
  • Try your deployment again