ExecuteBatchAsync() throws System.TimeoutException after running for some hours

1
votes

This is an interesting one: I have a console app running on a farm of XS Azure VMs sucking events (~1KB each) from EventHubs and storing them in Azure Table Storage. We got lots of events every day: ~10M. I am batching the events using ExecuteBatchAsync() - avg batch size is 10 events.The app runs quite well for a few hours, but after around 24hrs all nodes become sluggish and there are lots of "Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout." I have the following retry policy:

TableRequestOptions reqOptions = new TableRequestOptions()
{
  MaximumExecutionTime = TimeSpan.FromSeconds(60),
            RetryPolicy = new ExponentialRetry(TimeSpan.FromMilliseconds(300), 10),
            LocationMode = LocationMode.PrimaryOnly
        };

And this in my web.config:

expect100Continue="false" 
useNagleAlgorithm="false"
add address="*" maxconnection="24"

Any clues? Thanks in advance!

1
azureazure-storage

1 Answers

1
votes

Could you please try different VM size/disk mode (Premium, which does not work with the XS one)? It looks like you could hit the limits. https://azure.microsoft.com/en-us/documentation/articles/storage-scalability-targets/ Also i would not say that XS can be used for production or testing/debugging due to its nature.