Unable to connect to Azure Cosmos Db Account using Microsoft.EntityFrameworkCore.Cosmos - Response status code

Question

The CosmosDb provider is sending this message:

“Response status code does not indicate success: 503 Substatus: 0 Reason: (The request failed because the client was unable to establish connections to 3 endpoints across 1 regions. Please check for client resource starvation issues and verify connectivity between client and server.”

In my tests, it works (.net core 3.1):

Task.Run(async () =>
        {
            var endpoint = “test”;
            var masterKey = “test”;
            using (var client = new DocumentClient(new Uri(endpoint), masterKey))
            {
                //Insert new Document  
                Console.WriteLine("\r\n>>>>>>>>>>>>>>>> Creating Document <<<<<<<<<<<<<<<<<<<");
                dynamic candidato = new
                {
                    Id = 1,
                    Nome = "Test"
                };

                var document1 = await client.CreateDocumentAsync(
                    UriFactory.CreateDocumentCollectionUri("Test", "Test"),
                    candidato);

                Console.ReadKey();
            }

        }).Wait();

It does not:

            Task.Run(async () =>
            {
                using (var context = new StudentsDbContext())
                {
                    context.Add(new FamilyContainer(2, "Test"));
                    await context.SaveChangesAsync();
                }

            }).Wait();

public class FamilyContainer
{
    public int Id { get; set; }
    public string Nome { get; set; }

    public FamilyContainer(int id, string nome)
    {
        Id = id;
        Nome = nome;
    }

}

public class StudentsDbContext : DbContext
{
    public DbSet<FamilyContainer> FamilyContainer { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        optionsBuilder.UseCosmos(
           "test",
           "test",
           "FamilyDatabase",
           options =>
           { }
         );
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<FamilyContainer>(x =>
        {
            x.ToContainer("FamilyContainer");
        });
    }
}

Packages

Can anyone help me? Thanks

fail: Microsoft.EntityFrameworkCore.Update[10000] An exception occurred in the database while saving changes for context type '...'. Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: Maximum number of retries (6) exceeded while executing database operations with 'CosmosExecutionStrategy'. See inner exception for the most recent failure. ---> Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: 503 Substatus: 0 Reason: (Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.ActivityId: 07fbf539-0d44-4e5a-89d0-cd46838ee605, {"RequestStartTimeUtc":"2020-02-21T16:34:09.1834993Z","RequestEndTimeUtc":"2020-02-21T16:34:41.3484203Z","RequestLatency":"00:00:32.1649210","IsCpuOverloaded":false,"NumberRegionsAttempted":1,"ResponseStatisticsList":[{"ResponseTime":"2020-02-21T16:34:11.5964152Z","ResourceType":2,"OperationType":0,"StoreResult":"StorePhysicalAddress: rntbd:.../, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: True, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused [0x0000274D]... --- End of inner exception stack trace ---

Answer 1

I was facing same issue.

What worked for me is changing ConnectionMode to ConnectionMode.Gateway while initializing CosmosClient like :

var options = new CosmosClientOptions() { ConnectionMode = ConnectionMode.Gateway };
var client = new CosmosClient(endpoint, key, options); 

For more details on refer :

https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.cosmosclientoptions?view=azure-dotnet

https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.connectionmode?view=azure-dotnet

Answer 2

TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused

This means that the Connection was refused.

• Either your Cosmos DB account has Firewall/VPN enabled and the application is not able to establish a connection due not not being in a whitelisted IP/Network : Try checking your account configuration.
• The environment you are executing the code is restricting connections (some corporate Firewall or network might be blocking port ranges): Try running the app in a different network, or use GatewayMode. If that works, then this is related to the network.
• The machine might be running low on sockets or high on CPU.

Answer 3

My RCA for this is: Cosmos Partitions where served by individual processes on CosmosDB, each partition serving process has it's own TCP port. When client connects to 443 (Using TCP Direct Mode), CosmosDB Proxy sends partition ports back to client so that client can talk to server-partitions in parallel. Partition ports are random (11000 upwards afaik). Normal company firewall would allow outbound 443 (connection to cosmos works) but blocks the outbound random ports. So at the end, access fails. Workarounds:

1. Open firewall
2. Use Gateway Mode. This uses https/443 only by forwarding internally instead of redirecting to other ports.

Answer 4

It is because Entity framework has a default connection mode of Direct. It worked for me after overriding it to Gateway.

    {
        optionsBuilder.UseCosmos(
           "test",
           "test",
           "FamilyDatabase",
           options =>
           { options.ConnectionMode(ConnectionMode.Gateway); }
         );
    }