Azure CosmosDB: Getting Error during bulk deletion

Question

I am deleting bulk items and getting the below error on my local machine. I am using CosmosDB SDK 3.0.

An error occured trying to Initiailize CosmosBb. One or more errors occurred. (Response status code does not indicate success: ServiceUnavailable (503); Substatus: 0; ActivityId: befe13e5-172f-4930-953f-e24bf9b0a14a; Reason: (Service is currently unavailable. ActivityId: befe13e5-172f-4930-953f-e24bf9b0a14a, RequestStartTime: 2020-08-06T07:04:58.1807989Z, RequestEndTime: 2020-08-06T07:05:28.2986219Z, Number of regions attempted: 1 ResponseTime: 2020-08-06T07:04:59.1993537Z, StoreResult: StorePhysicalAddress: rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer18/partitions/a4cb495e-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: True, TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-08-06T07:04:59.1993537Z, activity ID: befe13e5-172f-4930-953f-e24bf9b0a14a, error code: ConnectFailed [0x0005], base error: socket error

Here is my delete method.

private async Task DeleteAllExistingSubscriptions(string userUUId)
        {
            var subscriptions = await _repository
                .GetItemsAsync(x => x.DistributionUserIds.Contains(userUUId), o => o.PayerNumber);

            if (subscriptions.Any())
            {
                List<Task> bulkOperations = new List<Task>();
                foreach (var subscription in subscriptions)
                {
                    bulkOperations.Add(_repository
                        .DeleteItemAsync(subscription.Id.ToString(), subscription.PayerNumber));
                }
                await Task.WhenAll(bulkOperations);
            }
        }

Cosmos Client:

private static void RegisterCosmosClient(IServiceCollection serviceCollection, IConfiguration configuration)
        {
            string cosmosDbEndpoint = configuration["CosmoDbEndpoint"];

            Ensure.ConditionIsMet(cosmosDbEndpoint.IsNotNullOrEmpty(),
                () => new InvalidOperationException("Unable to locate configured CosmosDB endpoint"));

            var cosmosDbAuthKey = configuration["CosmoDbAuthkey"];

            Ensure.ConditionIsMet(cosmosDbAuthKey.IsNotNullOrEmpty(),
                () => new InvalidOperationException("Unable to locate configured CosmosDB auth key"));

            serviceCollection.AddSingleton(s => new CosmosClient(cosmosDbEndpoint, cosmosDbAuthKey,
                new CosmosClientOptions { AllowBulkExecution = true }));
        }

my guess is to many tying to happen at the same time. i assume _repository has a shared connection... what is the max sim operations per connection. for example with EF to sql you cant do multiple transactions on one DbInstance... so maybe each Task needs its own? but that doesnt seem right either. Error msg not that helpful either. — Seabizkit

Matias Quaranta Matias Quaranta · Accepted Answer · 2020-08-06T15:57:08

The error is a client-side connectivity issue:

TransportException: A client transport error occurred: Failed to connect to the remote endpoint. 
(Time: 2020-08-06T07:04:59.1993537Z, 
activity ID: befe13e5-172f-4930-953f-e24bf9b0a14a, 
error code: ConnectFailed [0x0005], base error: socket error

Common cause is resource starvation:

The volume of data that you are sending through needs CPU to be processed, if the environment's CPU is not enough to process the volume of data, you will see CPU spikes (>70 - 100%) which will cause TCP connections to be delayed and would cause these issues. In this case, either reduce the volume of data or increase the available resources.
If running on the cloud, it could also be a [SNAT limitation](https://docs.microsoft.com/azure/cosmos-db/troubleshoot-dot-net-sdk#snat] or connection limit on the instance (depending on the service, connection limits vary).

Reference: https://docs.microsoft.com/azure/cosmos-db/troubleshoot-dot-net-sdk-request-timeout#troubleshooting-steps

Azure CosmosDB: Getting Error during bulk deletion

1 Answers