2
votes

When deploying an Azure Event Grid subscription, we started receiving a "Resource cannot be updated during provisioning" error starting end of last week, out of nowhere, with no change to the ARM template which had been working for several weeks.

The Continuous Deployment ARM Template is basically deploying the latest AppSettings of an Azure Functions App along with an Event Grid Topic Subscription. Here are the important steps from the CD, it fails at the first step, which executes the ARM template:

Continous Deployment Steps

Error Message from the Continuous Deployment in VSTS:

2018-04-16T15:42:12.8544806Z There were errors in your deployment. Error code: DeploymentFailed.
2018-04-16T15:42:12.8544806Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.
2018-04-16T15:42:12.8544806Z ##[error]Details:
2018-04-16T15:42:12.8544806Z ##[error]Conflict: {
  "status": "Failed",
  "error": {
    "code": "ResourceDeploymentFailure",
    "message": "The resource operation completed with terminal provisioning state 'Failed'.",
    "details": [
      {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"InvalidRequest\",\r\n    \"message\": \"Resource cannot be updated during provisioning\"\r\n  }\r\n}"
          }
        ]
      }
    ]
  }
} undefined
2018-04-16T15:42:12.8544806Z ##[error]Task failed while creating or updating the template deployment.

When looking at the Event Grid Activity Log in the Azure Portal, I found the following information, confirming that the subscription is failing:

{
  "authorization": {
    "action": "Microsoft.EventGrid/eventSubscriptions/write",
    "scope": "[REMOVED]"
  },
  "caller": "a11d0bff-196e-4967-a4be-7fa0c095d39f",
  "channels": "Operation",
  "claims": {
    "aud": "[REMOVED]"
  },
  "correlationId": "a7be5495-e9b8-47fd-a2be-9aa30b9ec54a",
  "description": "",
  "eventDataId": "005b7a10-b0d1-4281-9e44-26f33e4a3e39",
  "eventName": {
    "value": "EndRequest",
    "localizedValue": "End request"
  },
  "category": {
    "value": "Administrative",
    "localizedValue": "Administrative"
  },
  "eventTimestamp": "2018-04-16T16:26:31.5821162Z",
  "id": "[REMOVED]",
  "level": "Error",
  "operationId": "fea43c00-d46e-49d4-8c97-d78a9951aaaa",
  "operationName": {
    "value": "Microsoft.EventGrid/eventSubscriptions/write",
    "localizedValue": "Microsoft.EventGrid/eventSubscriptions/write"
  },
  "resourceGroupName": "[REMOVED]",
  "resourceProviderName": {
    "value": "Microsoft.EventGrid",
    "localizedValue": "Microsoft.EventGrid"
  },
  "resourceType": {
    "value": "Microsoft.EventGrid/eventSubscriptions",
    "localizedValue": "Microsoft.EventGrid/eventSubscriptions"
  },
  "resourceId": "[REMOVED]",
  "status": {
    "value": "Failed",
    "localizedValue": "Failed"
  },
  "subStatus": {
    "value": "",
    "localizedValue": ""
  },
  "submissionTimestamp": "2018-04-16T16:26:55.0807594Z",
  "subscriptionId": "[REMOVED]",
  "properties": {
    "statusCode": "BadRequest",
    "statusMessage": "{\"error\":{\"code\":\"InvalidRequest\",\"message\":\"Resource cannot be updated during provisioning\"}}",
    "serviceRequestId": "05c4b3e5-692a-4669-b8b4-c00a3a9cbd88"
  },
  "relatedEvents": [

  ]
}

Extract from ARM Template for the Event Grid subscription:

            {
        "apiVersion": "2017-05-10",
        "name": "nested-shared-rg",
        "type": "Microsoft.Resources/deployments",
        "resourceGroup": "[variables('globalSharedResourceGroupName')]",
        "properties": {
            "mode": "Incremental",
            "template": {
                "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
                "contentVersion": "1.0.0.0",
                "parameters": {},
                "variables": {},
                "resources": [
                    {
                        "name": "[concat(variables('notificationTopicName'), '/Microsoft.EventGrid/', variables('notificationTopicSubscriptionName'))]",
                        "type": "Microsoft.EventGrid/topics/providers/eventSubscriptions",
                        "apiVersion": "2017-06-15-preview",
                        "tags": {
                            "version": "[parameters('templateVersion')]",
                            "fullVersion": "[parameters('templateFullVersion')]",
                            "scope": "[parameters('webAppFctName')]",
                            "environment": "[parameters('environmentName')]",
                            "displayName": "Event Grid Subscription"
                        },
                        "properties": {
                            "destination": {
                                "endpointType": "WebHook",
                                "properties": {
                                    "endpointUrl": "[concat('https://', variables('webAppName') ,'.azurewebsites.net', variables('notificationEndPointPath'))]"
                                }
                            },
                            "filter": {
                                "includedEventTypes": [ "All" ],
                                "subjectBeginsWith": "",
                                "subjectEndsWith": "",
                                "subjectIsCaseSensitive": false
                            },
                            "labels": [ "" ]
                        }
                    }
                ],
                "outputs": {
                    "resourceGroupOutput": {
                        "type": "object",
                        "value": "[resourceGroup()]"
                    }
                }
            },
            "parameters": {}
        }
    }

Thank you! Simon

1
May I ask if your ARM template uses "dependsOn" to make the Grid subscription strictly follow the FunctionApp deployment? --> Modifying the ARM template to deploy only the Event Grid subscription as seen above gives the same error. Also note that the Event Grid subscription and the Functions App are not in the same resource group.Simon Luckenuik
I also tried deploying a previous Release of the package using same ARM template and the deployment now fails (was working last time it was deployed).Simon Luckenuik
have you solved this ?HariHaran
No, but it didn't happen to me for a long time, maybe something got fix on their side OR I've been lucky...Simon Luckenuik

1 Answers

0
votes

In this section:

https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#sample-script-to-stop-and-restart-triggers-and-clean-up

This line seems to cause an intermittent error if you have a lot of triggers (we have 36 event based triggers). Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_ -Force

By adding a short delay (5 seconds), it appears to avoid the error. i.e. "Start-Sleep -s 5" inside the foreach."