.NetCore App Engine Flex Deployment: Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation

Question

I am getting the error

Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation

while deploying .NetCore application to my App Engine. App Engine is currently in Asia South. I also created another project with App Engine at US East. On that I am getting error

ERROR: (gcloud.app.deploy) INTERNAL: Internal error encountered.

while deploying. Tried to deploy the Hello world application specified in GCP docs. It is also getting same issues while deploying. Am I missing something. I have updated BACKEND_SEVICES quota. After that only I am getting this issue. Also able to see versions as serving. But no Instances are active. I am not able to activate any other versions. It is also showing as Internal error.

This is my app.YAML

runtime: aspnetcore
env: flex
manual_scaling:
  instances: 1
resources:
  cpu: 1
  memory_gb: 0.5
  disk_size_gb: 10
beta_settings:
    cloud_sql_instances: "#########:#####-api-system=tcp:3306"

It was working fine till yesterday. I am getting this issue from today onwards.

Attaching Verbosity Logs

Updating service [default] (this may take several minutes)...|DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...|DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...\DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...\DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...-DEBUG: Operation [apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067] complete. Result: {
    "done": true,
    "error": {
        "code": 4,
        "message": "App Engine Flexible timed out while configuring resources, internal operation projects/**projectid**/regions/asia-south1/operations/700238de-3add-4281-80f8-6dc66948a715"
    },
    "metadata": {
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
        "createVersionMetadata": {
            "cloudBuildId": "38b57f34-3b23-4484-9d69-0c581995947f"
        },
        "endTime": "2020-04-23T23:40:50.335Z",
        "insertTime": "2020-04-23T23:29:10.164Z",
        "method": "google.appengine.v1.Versions.CreateVersion",
        "target": "apps/**projectid**/services/default/versions/20200424t045856",
        "user": "[email protected]"
    },
    "name": "apps/**projectid**/operations/c46a17bd-d4a8-468f-b40b-47a7fdd03067"
}
Updating service [default] (this may take several minutes)...failed.
DEBUG: (gcloud.app.deploy) Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation projects/**projectid**/regions/asia-south1/operations/700238de-3add-4281-80f8-6dc66948a715
Traceback (most recent call last):
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\calliope\cli.py", line 983, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\calliope\backend.py", line 807, in Run
    resources = command_instance.Run(args)
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\surface\app\deploy.py", line 111, in Run
    return deploy_util.RunDeploy(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 634, in RunDeploy
    deployer.Deploy(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 431, in Deploy
    self.api_client.DeployService(new_version.service, new_version.id,
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\appengine_api_client.py", line 204, in DeployService
    return operations_util.WaitForOperation(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\operations_util.py", line 307, in WaitForOperation
    completed_operation = waiter.WaitFor(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 261, in WaitFor
    operation = PollUntilDone(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 322, in PollUntilDone
    operation = retryer.RetryOnResult(
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\core\util\retry.py", line 229, in RetryOnResult
    if not should_retry(result, state):
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 320, in _IsNotDone
    return not poller.IsDone(operation)
  File "C:\Google\CloudSDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\operations_util.py", line 182, in IsDone
    raise OperationError(requests.ExtractErrorMessage(
googlecloudsdk.api_lib.app.operations_util.OperationError: Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation projects/**projectid**/regions/asia-south1/operations/700238de-3add-4281-80f8-6dc66948a715
ERROR: (gcloud.app.deploy) Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation projects/**projectid**/regions/asia-south1/operations/700238de-3add-4281-80f8-6dc66948a715

App.yaml

runtime: aspnetcore
env: flex
manual_scaling:
  instances: 1
resources:
  cpu: 1
  memory_gb: 0.5
  disk_size_gb: 10
beta_settings:
    cloud_sql_instances: "***projectid***:asia-south1:***dbname***=tcp:3306"

Thanks.

Joss Baron Joss Baron · Accepted Answer · 2020-04-23T21:55:54

It is recommended to get more detail in this cases. You can deploy with: gcloud app deploy --verbosity=debug, just to see if you can retrieve more information about this.

It seems this could be due to some reasons. I recommend to check:

In some cases is useful to run this command $ gcloud config set app/trigger_build_server_side false in order to force that the building will be managed on the client side rather than the server side (which could be changed in somehow), before deploying. Try this command and then deploy again.
Possible changes in the configuration of your app. Take a look into your app.yaml, check for different values. Comparing the version of Apr 22 and that the one of today. Is it possible for you to deploy the version of yesterday?
The quotas in your project associated to App Engine and Compute Engine. Remember that, in the end, App Engine Flex runs in VMs. Here is a similar issue, related with the quota: In-use IP addresses. If you see any quota exhausted, you can request a Quota increase.
In some cases, this can also be related to an internal issue, in such case you should create a case in GCP. (lets keep this as final option)
Also, could you pleas share relevant detailed logs about this? Please, be sure of not sharing any sensitive information

Please let me know your findings with this. Also it would be great if you can share the app.yaml of both versions (working and not) in case the aforementioned does not work.

.NetCore App Engine Flex Deployment: Error Response: [4] App Engine Flexible timed out while configuring resources, internal operation

1 Answers