The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue

Question

I'm using a Microsoft azure service bus queue to process calculations and my program runs fine for a few hours but then I start to get this exception for every message that I process from then on. I have no clue where to start since everything runs fine for the first few hours. My code seems to be accurate as well. I will post the method where I handle the azure service bus message.

public static async Task processCalculations(BrokeredMessage message)
    {
        try
        {
            if (message != null)
            {
                if (connection == null || !connection.IsConnected)
                {
                    connection = await ConnectionMultiplexer.ConnectAsync("connection,SyncTimeout=10000,ConnectTimeout=10000");
                    //connection = ConnectionMultiplexer.Connect("connection,SyncTimeout=10000,ConnectTimeout=10000");
                }

                cache = connection.GetDatabase();

                string sandpKey = message.Properties["sandp"].ToString();
                string dateKey = message.Properties["date"].ToString();
                string symbolclassKey = message.Properties["symbolclass"].ToString();
                string stockdataKey = message.Properties["stockdata"].ToString();
                string stockcomparedataKey = message.Properties["stockcomparedata"].ToString();

                var sandpTask = cache.GetAsync<List<StockData>>(sandpKey);
                var dateTask = cache.GetAsync<DateTime>(dateKey);
                var symbolinfoTask = cache.GetAsync<SymbolInfo>(symbolclassKey);
                var stockdataTask = cache.GetAsync<List<StockData>>(stockdataKey);
                var stockcomparedataTask = cache.GetAsync<List<StockMarketCompare>>(stockcomparedataKey);

                await Task.WhenAll(sandpTask, dateTask, symbolinfoTask,
                    stockdataTask, stockcomparedataTask);

                List<StockData> sandp = sandpTask.Result;
                DateTime date = dateTask.Result;
                SymbolInfo symbolinfo = symbolinfoTask.Result;
                List<StockData> stockdata = stockdataTask.Result;
                List<StockMarketCompare> stockcomparedata = stockcomparedataTask.Result;

                StockRating rating = performCalculations(symbolinfo, date, sandp, stockdata, stockcomparedata);

                if (rating != null)
                {
                    saveToTable(rating);
                    if (message.LockedUntilUtc.Minute <= 1)
                    {
                        await message.RenewLockAsync();
                    }
                    await message.CompleteAsync(); // getting exception here
                }
                else
                {
                    Console.WriteLine("Message " + message.MessageId + " Completed!");
                    await message.CompleteAsync();
                }
            }
        }
        catch (TimeoutException time)
        {
            Console.WriteLine(time.Message);
        }
        catch (MessageLockLostException locks)
        {
            Console.WriteLine(locks.Message);
        }
        catch (RedisConnectionException redis)
        {
            Console.WriteLine("Start the redis server service!");
        }
        catch (MessagingCommunicationException communication)
        {
            Console.WriteLine(communication.Message);
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
            Console.WriteLine(ex.StackTrace);
        }
    }

UPDATE: I check the time until the lock expiration and I call lock renew if it needs it but it renews the lock with no errors but I'm still getting this exception.

timeLeft = message.LockedUntilUtc - DateTime.UtcNow;
  if (timeLeft.TotalMinutes <= 2)
                    {
                        //Console.WriteLine("Renewed lock! " + ((TimeSpan)(message.LockedUntilUtc - DateTime.UtcNow)).TotalMinutes);
                        message.RenewLock();
                    }

catch (MessageLockLostException locks)
        {
            Console.WriteLine("Delivery Count: " + message.DeliveryCount);
            Console.WriteLine("Enqueued Time: " + message.EnqueuedTimeUtc);
            Console.WriteLine("Expires Time: " + message.ExpiresAtUtc);
            Console.WriteLine("Locked Until Time: " + message.LockedUntilUtc);
            Console.WriteLine("Scheduled Enqueue Time: " + message.ScheduledEnqueueTimeUtc);
            Console.WriteLine("Current Time: " + DateTime.UtcNow);
            Console.WriteLine("Time Left: " + timeLeft);
        }

All I know so far is that my code runs fine for awhile and the renew lock gets called and works but I'm still getting the lock exception and inside that exception, I output the timeleft and it keeps increasing the time difference as the code runs which makes me believe that the time until lock expiration is not being changed somehow?

@FyodorSoikin No I eventually had to give up and work on a different method to do the same thing. From what I could tell, it is a bug in the api but no one from Microsoft ever responded to my posts — DarthVegan
By any chance could any of your individual messages take longer than 60 seconds to process? — DalSoft
@DalSoft I would renew the lock after the intensive calculation was finished and right before it processed the message — DarthVegan
@user3610374 have you tried setting the LockDuration to 5 minutes (which is the max setting), the default is 60 seconds and this exception will be thrown if 60 seconds pass before lock renewals. — DalSoft

Jeffrey LeCours Jeffrey LeCours · Accepted Answer · 2016-07-01T15:47:47

I spent hours trying understand why I was getting a MessageLockLostException. The reason for me was due to AutoComplete defaulting to true.

If you're going to call messsage.Complete() (or CompleteAsync()) then you should instantiate an OnMessageOptions object, set AutoComplete to false, and pass it into your OnMessage call.

var options = new OnMessageOptions();
options.AutoComplete = false;

client.OnMessage(processCalculations, options);

The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue

5 Answers

ServiceBus Queue or Topic-subscription config:

ServiceBusClient options config:

ServiceBusProcessor options config: