"Unexpected Response Code for Operation: 0" when executing Azure Table Storage batch delete

Question

I'm using version 4.3.0 of the Windows Azure Storage libraries for .NET. In my ATS repository class, I have a couple of batch delete methods that look like this:

public async Task DeleteAsync(IEnumerable<T> entities)
{
    await ExecuteAsBatch(entities, (batch, entity) => batch.Delete(entity));
}

private async Task ExecuteAsBatch(IEnumerable<T> entities, Action<TableBatchOperation, T> batchAction)
{
    var byPartition = entities.GroupBy(x => x.PartitionKey).ToList();

    await byPartition.ForEachParallel(async group =>
    {
        // A maximum of 100 actions are allowed per batch job
        var segments = group.ToList().ToSegmentedList(100);
        await segments.ForEachParallel(async segment =>
        {
            var batch = new TableBatchOperation();
            foreach (var entity in segment)
            {
                batchAction(batch, entity);
            }
            await Table.ExecuteBatchAsync(batch);
        }, 10);
    }, 10);
}

In other places in my code, that DeleteAsync() method works correctly. However, in one particular place, I get this error message when executing the batch:

Unexpected Response Code for Operation: 0

Here's the call site:

private async Task MergeAtsOrganizationUserEvents(int organizationId, IEnumerable<CustomerUserEvent> fromEvents, CustomerUser to)
{
    var toDelete = (await fromEvents.SelectParallel(async fromEvent =>
    {
        var pkey = AtsOrganizationUserEventByMinute.GetPartitionKey(organizationId, fromEvent.OccurredOn);
        var rkey = AtsOrganizationUserEventByMinute.GetRowKey(fromEvent.OccurredOn, fromEvent.CustomerUserEventId);
        return await Ats.OrganizationUserEventByMinute.FindByPartitionRowAsync(pkey, rkey);
    })).Where(x => x != null).ToList();

    var toInsert = toDelete
        .Select(x => AtsOrganizationUserEventByMinute.FromBase(x.OrganizationId, x.OccurredOn, x.CookieId,
            to.CustomerUserId, x))
        .ToList();

    try
    {
        await Ats.OrganizationUserEventByMinute.UpsertAsync(toInsert);
        await Ats.OrganizationUserEventByMinute.DeleteAsync(toDelete);
    }
    catch (Exception ex)
    {
        _logger.Error("Unable to merge {0} AtsOrganizationEvents for org {1}, to customer user {2}: {3}",
            toInsert.Count, organizationId, to.CustomerUserId, ex.CompleteMessage());
        throw;
    }
}

The UpsertAsync() method above succeeds, but the DeleteAsync() fails. Note that it fails to delete precisely the same entities that FindByPartitionRowAsync() retrieved from the table, so I can't quite imagine how it could have anything to do with malformed entities or anything of that ilk.

Here's an example of one of the "toDelete" objects (in JSON format):

{  
   "CookieId":null,
   "CustomerUserId":185766,
   "CustomerUserEventId":3568687,
   "OrganizationId":4190,
   "EventName":"event1",
   "SessionId":null,
   "OccurredOn":"2014-10-20T18:17:09.9971379Z",
   "UrlId":null,
   "Url":null,
   "ReferrerUrlId":null,
   "ReferrerUrl":null,
   "IsSynthetic":false,
   "IpAddress":null,
   "PartitionKey":"4190.2014.10.20",
   "RowKey":"18.17.3568687",
   "Timestamp":"2014-10-20T18:17:11.237+00:00",
   "ETag":"W/\\"   datetime'2014-10-20T18%3A17%3A11.237Z'\\""
}

Azure Storage error messages are notoriously and spectacularly unhelpful, and Googling has returned nothing about batch deletes failing with this particular error.

This fails both when using local development storage and in production.

Any thoughts?

Hi Ken, would you mind sharing your .ForEachParallel() and .ToSegmentedList() code it's something that looks very handy. Thx. — Tinus Heystek
Sure. See gist.github.com/smithkl42/ca5b98e909952eddef1e3a076c0dd91d and gist.github.com/smithkl42/36ca70a63fe1a3bbf2d03a4d0bbe46c2. — Ken Smith

Veena Udayabhanu - MSFT Veena Udayabhanu - MSFT · Accepted Answer · 2014-10-21T01:21:41

'Unexpected Response Code for Operation: 0' basically means that the first operation in the batch failed. The index of the failed operation is returned in the error thrown so it makes it easier for users to go and change the specific operation in the batch that failed.

You can get more information about the request that failed and the error by catching the StorageException and checking:

exception.RequestInformation.HttpStatusCode
exception.RequestInformation.ExtendedErrorInformation.ErrorCode
exception.RequestInformation.ExtendedErrorInformation.ErrorMessage

The same information is also available in the OperationContext's last result if you use an OperationContext to track the request and use suitable method overloads that take in the OperationContext.

We will look at changing the error message in the future so it is less confusing. Thanks for the feedback!

"Unexpected Response Code for Operation: 0" when executing Azure Table Storage batch delete

3 Answers