Checkpointing position of a Consumer Group in Azure Event Hubs with AMQP

Question

I am developing some code to process events off of an Azure Event Hub using AMQP with the Apache Qpid library. One of the things I'm noticing is that when my application restarts, all messages are re-read from the consumer group / partition.

My assumption is that my consumer is not checkpointing as it should (based on https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features#event-consumers), but I am not sure what I options would need to be set on the JMS consumer to do this.

My current connection code (prior to attaching message listeners) looks something like this:

    final ConnectionFactory factory = new JmsConnectionFactory(uri);
    final Connection connection = factory.createConnection();
    connection.start();

    final Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

Is there something I need to do in terms of URL options to cause checkpointing to occur?

Michael Michael · Accepted Answer · 2018-09-18T19:15:30

The short answer to this question is that AMQP has no notion of checkpointing built in and, by extension, neither does JMS. The result of this is that every time an application that reads via AMQP starts up, it will begin reading from the beginning of the event stream and reprocess everything.

If the application is developed properly (because an intentional rewind is quite possible), this shouldn't cause a functional problem, but it does have the potential to be very wasteful of resources. In the end I settled on using Microsoft's Event Hub client for Java, which has checkpointing support built in.

I sketched this out in some sample code on my github page, comparing https://github.com/michaeljmcd/eventhub-qpid-example and https://github.com/michaeljmcd/eventhub-client-example

Checkpointing position of a Consumer Group in Azure Event Hubs with AMQP

1 Answers