Azure Blob storage .Net client library and TCP connections over time

Question

We are using Azure blob storage to store incoming messages from a data feed (we receive about 1.5GB a day of data) and then processing them which is triggered via a message queue (rabbitmq). Here is how the setup looks like:

Producer -> Store XML file in Azure blob -> Publish blob address to queue

Consumer -> Read the blob address from the queue -> Download blob in memory

And here is the Download blob method that is executed for each message:

private string GetBlobText(string containerName, string blobName)
{
    // Parse the connection string and return a reference to the storage account.
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(Settings.Default.DatafeedStorageConnString);
    CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
    CloudBlobContainer container = blobClient.GetContainerReference(containerName);
    CloudBlockBlob blockBlob = container.GetBlockBlobReference(blobName);
    return blockBlob.DownloadText(Encoding.UTF8);
}

This pipeline is running at a fairly frequent pace and hence we see that over time (some weeks) the program starts receiving Socket errors. Here is the trace:

Microsoft.WindowsAzure.Storage.StorageException: Unable to connect to the remote server ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 40.68.232.24:443
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 695
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 604
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadRangeToStream(Stream target, Nullable`1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 675
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadToStream(Stream target, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 234
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.DownloadText(Encoding encoding, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 1279

This seems like an issue with not handling connections efficiently, we had tried reusing blobClient, but that has not helped us. What shall we look into further to solve this?

Bruce Chen Bruce Chen · Accepted Answer · 2017-09-28T07:08:21

Microsoft.WindowsAzure.Storage.StorageException: Unable to connect to the remote server ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 40.68.232.24:443 (blob.am5prdstr02a.store.core.windows.net)

Per my understanding, it may be TCP/IP Port Exhaustion. I assumed that you could use Netstat to query the state of your network connections and the ports you are using. For more details, you could refer to here for explanation and ways to diagnose this issue.

If the Port Exhaustion occurs in your client computer, I assume you could increase the the upper range of ephemeral ports for client TCP/IP socket connections and reduce the client TCP/IP socket connection timeout, for more details, you could refer to here. Also, you could scale your client application into multiple instances among different computers.

Azure Blob storage .Net client library and TCP connections over time

1 Answers