Re-trigger Azure Function with abandonded Service Bus messages

Question

I have a Service Bus topic that has 400,000 messages pushed to it from an Azure Function. I have a second Azure Function that receives the messages as a trigger. When the 2nd Function ran, it successfully processed 98% of the messages. It left me with roughly 8,000 messages that failed. Either from an exception, or from my code, the message was abandoned. I've now got 8,000 messages sitting in a subscriber of the topic, that I can't get the Function to re-try processing.

The subscriber was originally set up to only allow 1 delivery of the message. I had done it that way because I was seeing the same message get processed multiple times while debugging. I wasn't sure if that was a side-effect of debugging in Visual Studio locally, or if messages would get sent to a Function multiple times. I assume this is what caused those messages to not have the Function re-run after they were abandoned (if that even is supported?).

I've changed the subscriber delivery count to 5, hoping that it would re-deliver the message to the Function. It didn't. What do I need to do now to get the remaining 8,000 messages to trigger the 2nd Function again?

Worst case is that I can delete the subscription, re-create the subscription and run the first Function over again, which should just publish the data associated with the original missing 8,000 messages (I have logic to handle missing data in the 1st Function). That would cause the 2nd Function to re-fire. I'd like to figure out how to handle it though when the subscription has the orphaned messages like this, as I'll experience this when I promote to production. I want this to be automated and not have to deal with manually cleaning up and re-running the process a 2nd time.

The following shows that there aren't any messages left in the Topic, but they exist still in the subscription.

I wrote the Functions in C# Functions 1.0 on Full Framework.

you can get the messages from the deadletter or poison queue. — Thomas
So I cant get the original Service bus triggered Azure Function to run again? I have to write a utility app to go get those messages? — Johnathon Sullinger
On your azure dashboard, could you see this messages in the deadletter or poison queue ? if so yes you can get them — Thomas
No, the message count for everything is 0 under the Topic. The subscription has the 8,000 messages but the Topic is empty. — Johnathon Sullinger
I updated my OP with a screenshot of the topic for reference. — Johnathon Sullinger

Thomas Thomas · Accepted Answer · 2018-04-04T03:13:31

If the MaxDeliveryCount is reach the message will be sent to a deadletter queue.
You can think about a subscription as a sub-queue so you can still access these messages.

Check this post on how to access deadletter queues:

How do you access the dead letter sub-queue on an Azure subscription?

First thing is your scenario will be to increase the MaxDeliveryCount.

Then you can create a function that will be triggered every time a message arrived in the deadletter queue/subscription:

 [FunctionName("my-subscription-failure")]
 public static async Task RunFailure(
        [ServiceBusTrigger("%MytopicName%", "%MySubscription%/$DeadLetterQueue", Connection = "MyconnectionString")] BrokeredMessage message,
        [ServiceBus( "%MytopicName%", AccessRights.Send, Connection = "MyconnectionString")] ICollector<BrokeredMessage> queueBinding,
        TraceWriter log, ExecutionContext context)
{
    // fist check if the message can be resent (it could be a message that can't be replay)
    ...

    // If ok, resent the message
    queueBinding.Add(message);
}

Re-trigger Azure Function with abandonded Service Bus messages

1 Answers