EDIT: I've just tested this again pointing at a temporarily created table on real Azure, and that's worked fine whereas I could repeatedly get this to happen with the local Emulator. So I suspect this is a bug with the Emulator?
Anyone know anything about that?
Extremely similar to Getting "Entity already exists" error writing aggregates to Azure Table Storage (with Azure Function) except that I've logged out all my records and I'm very confident that my code is generating unique Partition/Row Keys, which don't already exist in the Storage Table.
My insert code looks like this:
public static async Task BatchInsert<TEntity>(this CloudTable table, IEnumerable<TEntity> entities)
where TEntity : TableEntity
{
var entityPartitionGroups = entities.GroupBy(e => e.PartitionKey).ToList();
await Task.WhenAll(entityPartitionGroups.Select(async partitionGroup =>
{
var batches = partitionGroup.Batch(BatchSize).Select(batch => batch.ToList()).ToList();
foreach (var batchToInsert in batches)
{
var batchOperation = new TableBatchOperation();
foreach (var tableEntity in batchToInsert)
{
batchOperation.Insert(tableEntity);
}
try
{
var entitiesAsStrings = batchToInsert.Select((entity, i) => $"{i}: {entity.PartitionKey}, {entity.RowKey}").StringJoin(";");
Debug.WriteLine("Writing: " + entitiesAsStrings);
await table.ExecuteBatchAsync(batchOperation);
Debug.WriteLine("Wrote: "+entitiesAsStrings);
}
catch (StorageException ex)
{
// break point here.
throw new AzureTableBatchInsertException(batchToInsert, ex);
}
}
}));
}
- I'm writing each Partition in parallel, to try to increase insert performance.
- Within each Partition, I write all the records in series; I use
.ExecuteBatchAsync()to write batches of 100 records at a time. - When I run this, with the debugger enabled, and a break point is set. I've also wiped the target table, which is in the local Storage emulator.
- Nothing else is running that could talk to the table.
So I'm starting from a blank slate, and looking at the logs and everything confirms that the records I'm putting in are unique, but I'm still getting the "Entity already exists" error. (It's accusing element [0] of being the problem, so it thinks the entire batch has already been written.)
When I hit the break point, and look the in the storage emulator (using MS Azure Storage Explorer) the records ARE present, despite an initially blank DB, and only 1 log record of attempting to write the record.
It seems a lot like sometimes .ExecuteBatchAsync() retries a batch, despite having successfully written it?
What's going on here and how do I stop it from incorrectly throwing?
Happy to post the relevant portions of the Debug Output, but there's nothing at all interesting going on in it.
