Cosmos DB Change Feed Trigger Azure Function: Lease Lost Exception

Question

Below exception logged in the Application insights for Cosmos DB change feed trigger azure function:

Microsoft.Azure.Documents.ChangeFeedProcessor.Exceptions.LeaseLostException

[{"severityLevel":"Error","outerId":"0","message":"The lease was lost.","parsedStack":[{"assembly":"Microsoft.Azure.Documents.ChangeFeedProcessor, Version=2.2.6.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.Documents.ChangeFeedProcessor.LeaseManagement.DocumentServiceLeaseStoreManager+d__16.MoveNext","level":0,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":1,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":2,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":3,"line":0},{"assembly":"Microsoft.Azure.Documents.ChangeFeedProcessor, Version=2.2.6.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.Documents.ChangeFeedProcessor.PartitionManagement.PartitionController+d__9.MoveNext","level":4,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":5,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":6,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":7,"line":0},{"assembly":"Microsoft.Azure.Documents.ChangeFeedProcessor, Version=2.2.6.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.Documents.ChangeFeedProcessor.HealthMonitoringPartitionControllerDecorator+d__3.MoveNext","level":8,"line":0}],"type":"Microsoft.Azure.Documents.ChangeFeedProcessor.Exceptions.LeaseLostException","id":"517071"}

Cosmos DB Change Feed Trigger Azure Function:

public static class NotificationChangeFeed
    {
        [FunctionName(nameof(NotificationChangeFeed))]
        public static async Task Run([CosmosDBTrigger(
            databaseName: CosmosDBConstants.DataBaseName,
            collectionName: CosmosDBConstants.NotificationContainer,
            ConnectionStringSetting = CosmosDBConstants.ConnectionStringName,
            CreateLeaseCollectionIfNotExists = true,
            LeaseCollectionName = CosmosDBConstants.LeaseConainer)]IReadOnlyList<Document> input,
            [Inject] ILoggingService loggingService,
            [Inject] IEmailProcessor emailProcessor)
        {
            var logger = new Logger(loggingService);

            try
            {
                if (input != null && input.Count > 0)
                {
                    foreach (Document document in input)
                    {
                        string requestBody = document.ToString();
                        var notification = requestBody.AsPoco<Notification>();

                        await emailProcessor.HandleEmailAsync(notification, logger);
                        logger.Info($"Email Notification sent successfully for file name: {document.Id}");
                    }
                }
            }
            catch (Exception ex)
            {
                logger.Error($"Unable to process Documents for Email Notification for Files: {input?.Count}", ex,
                    nameof(NotificationChangeFeed));
                throw;
            }
        }
    }

Answer 1

This error means that lease is lost, that would typically happen when it is taken by another host. Other cases: communication failure, number of retries reached, lease not found.

• The preferred method to handle leases is to use automatic Lease Checkpoint implementation (if at all you are using manual checkpointing).
• You can also check if Lease collection is present or not.

Answer 2

LeaseLost is a normal signal that represents that the current instance was owning a lease, but due to load balancing (maybe another instance came up or the number of instances was changing) it got taken by another host. This is expected during initialization (first time starting up a set of instances) or during rebalancing due to number of instances changing.

There is nothing the user is expected to do, as this is part of the normal life cycle. The lease is now processed by another instance and that instance will read the changes happening in the partition that lease is scoped to.