Can I change the Azure Function Event Hub trigger starting checkpoint?

Question

When I stand up a new Azure Function listening to our event hub, with a new consumer group, it gets all the messages that have ever been sent to the event hub, even from months ago somehow. I thought setting the message retention on the hub to one day would limit that, but apparently not. Anyway, I really only want the new Azure Function to receive events from the initial deployment or testing point onward, however I don't know how I can specify that initial starting checkpoint, since that stuff is magically done in storage by AF.

If I'm doing the hub logic manually I can specify this checkpoint and start from an arbitrary time, such as the current time. Is there a way to replicate this with an Azure Function?

Answer 1

The good news is Yes, you can change the checkpoint. There is not much magic there: the checkpoints are stored in Blob Storage -> azure-webjobs-eventhubs container in blobs with names like yournamespace.servicebus.windows.net/yourhub/$Default/X, where $Default is consumer group name and X is partition number.

Here is an example of such blob:

{"PartitionId":"0","Owner":null,"Token":null,"Epoch":567,"Offset":"14992",
 "SequenceNumber":156}

The bad news is that you need to know your offset for the moment you want to start at, and you probably don't know it. And then, you need to go ahead and modify the blobs either manually or with custom code. And you should do that before the Function App starts.

It's up to you to decide whether you can/want to do that. AFAIK, there's no easy and official way to accomplish this.