Azure function exception - Error writing logs to table storage: Microsoft.Azure.Cosmos.Table.StorageException

Question

I have an azure function that communicates with blob storage to read the last sync date time and then based on that, reading the data from the Cosmos database.

Few interactions work fine and randomly throwing the following exception in some cases

"Error writing logs to table storage: Microsoft.Azure.Cosmos.Table.StorageException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. ---> System.Net.Http.HttpRequestException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. ---> System.Net.Sockets.SocketException (10055): An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor.ExecuteAsync[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) --- End of inner exception stack trace --- at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor.ExecuteAsync[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) at Microsoft.Azure.WebJobs.Logging.Utility.SafeExecuteAsync(CloudTable table, TableBatchOperation batch) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Logging\Internal\Utility.cs:line 178 at Microsoft.Azure.WebJobs.Logging.Utility.WriteBatchAsync[T](ILogTableProvider logTableProvider, IEnumerable1 e1) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Logging\Internal\Utility.cs:line 268 at Microsoft.Azure.WebJobs.Logging.LogWriter.FlushTimelineAggregateAsync(Boolean always) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Logging\Internal\LogWriter.cs:line 265 at Microsoft.Azure.WebJobs.Logging.LogWriter.FlushCoreAsync() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Logging\Internal\LogWriter.cs:line 316 Request Information RequestID: RequestDate: StatusMessage: ErrorCode:"

Can any one advise me how I can fix it?

Answer 1

Since you did not share any relevant piece of code from your function, my answer would be bit of generic. It seems to be due to socket exhaustion issue. While there is an inconclusive similar issue in github having some more details, a few good pointers to eliminate any low hanging fruits would be (I highly recommend you to read):

• Manage connections in Azure Functions
• Improper Instantiation antipattern

The major one I would particularly advise you to ensure shared connection reuse for Cosmos (or Table) client, HttpClient etc in a static private instance outside method.

Static clients

To avoid holding more connections than necessary, reuse client instances rather than creating new ones with each function invocation. We recommend reusing client connections for any language that you might write your function in. For example, .NET clients like the HttpClient, DocumentClient, and Azure Storage clients can manage connections if you use a single, static client.

Here are some guidelines to follow when you're using a service-specific client in an Azure Functions application:

• Do not create a new client with every function invocation.
• Do create a single, static client that every function invocation can use.
• Consider creating a single, static client in a shared helper class if different functions use the same service.

Example:

using Microsoft.Azure.Documents.Client;

private static Lazy<DocumentClient> lazyClient = new Lazy<DocumentClient>(InitializeDocumentClient);
private static DocumentClient documentClient => lazyClient.Value;

private static DocumentClient InitializeDocumentClient()
{
    // Perform any initialization here
    var uri = new Uri("example");
    var authKey = "authKey";
    
    return new DocumentClient(uri, authKey);
}

// Function
public static async Task Run(string input)
{
    Uri collectionUri = UriFactory.CreateDocumentCollectionUri("database", "collection");
    object document = new { Data = "example" };
    await documentClient.UpsertDocumentAsync(collectionUri, document);
    
    // Rest of function
}