Writing to CloudAppendBlob: The remote server returned an error: (409) Conflict & ProcessExpectedStatusCodeNoException

Question

I am trying to understand what causes the The remote server returned an error: (409) Conflict Exception.

I am writing lines to an AppendBlob by acquiring a lease, appending the text, and breaking the lease, like so. The error only happens every now and then, so I am curious what scenario would cause it to happen. My suspicion would be something may happen if multiple users hit the endpoint where this code executes and somehow to a call to acquire a lease happens at the same time.

// Get the AppendBlob reference
CloudAppendBlob blob = ….

// Acquire a lease on the blob until a call to breaking the lease is made
string leaseId = Guid.NewGuid().ToString();
await blob.AcquireLeaseAsync(null, leaseId);

// Append a new entry to the blob
await blob.AppendTextAsync(
       textToBeAppended,
       Encoding.UTF8,
       new AccessCondition { LeaseId = leaseId },
       new BlobRequestOptions(),
       new OperationContext());

// break the lease on the blob
await blob.BreakLeaseAsync(null);

The Failed Method is Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException.

I see from telemetry that a call to PUT on the blob initially fails with the 409 status code and the exception above, but immediately after a successful PUT call with a 202 result code occurs. I'm assume Azure Storage is doing a retry, however I do not see the line of text in the file it was supposed to be appended to.

Here is the full call stack if that is helpful, I am not sure what the meaning of ProcessExpectedStatusCodeNoException is from this stack trace.

Microsoft.WindowsAzure.Storage.StorageException: at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync (Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.csMicrosoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 50) at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions+<>c__DisplayClass2`1.b__0 (Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.csMicrosoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 69) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at Microsoft.Community.Data.Providers.BlobStorageProvider+d__1.MoveNext (Microsoft.Community.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Community.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\BA\813\s\Microsoft.Community\Microsoft.Community.Data\Providers\BlobStorageProvider.csMicrosoft.Community.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 67) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at Microsoft.Community.Data.Providers.BlobStorageProvider+d__1.MoveNext (Microsoft.Community.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Community.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\BA\813\s\Microsoft.Community\Microsoft.Community.Data\Providers\BlobStorageProvider.csMicrosoft.Community.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 72) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at Microsoft.Community.Web.Repositories.AzureMLRecommendationsRepository+d__6.MoveNext (Microsoft.Community.Web, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Community.Web, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\BA\813\s\Microsoft.Community\Microsoft.Community.Web\Repositories\AzureMLRecommendationsRepository.csMicrosoft.Community.Web, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 95)

Inner exception System.Net.WebException handled at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync: at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException (Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Shared\Protocol\HttpResponseParsers.Common.csMicrosoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 54) at Microsoft.WindowsAzure.Storage.Blob.CloudBlob+<>c__DisplayClass39.b__38 (Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.csMicrosoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 3830) at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse (Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.csMicrosoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 299)

Zhaoxing Lu Zhaoxing Lu · Accepted Answer · 2018-08-24T01:47:55

I'm wondering why you need to lease the blob and then append the text to the blob. If there are multiple writers simultaneously, it's EXPECTED that you'll meet (409) Conflict in your code.

As you can see in the official doc, acquiring lease (B) when a blob is still being leased by lease (A) will encounter 409 error:

Basically, blob lease is to achieve exclusive write access on a blob, it's inappropriate to use blob lease if you want to append against a blob from multiple writers simultaneously.

What are you afraid of if the AcquireLease and BreakLease operations are removed from your code? To be honest, I don't see any harm.

Writing to CloudAppendBlob: The remote server returned an error: (409) Conflict & ProcessExpectedStatusCodeNoException

2 Answers