Worker role - EventProcessorHost CheckPoint seems cause 409 Conflict Azure Storage

Question

I have a cloud service and an Event hub with 2 partitions. My cloud service is a worker role that writes into Azure Storage. It writes all messages that receives from Event Hub.

With Azure Emulator, my worker role works fine, it writes to the Azure Storage in production (not dev, so it's the same azure storage).

When I pushed my worker role to the cloud service, I receive this kind of error from Application Insight (100% Successful call: false). With IntelliTrace logs, I got a 409 Conflict.

I tried to remote debugging but It's so slow that I will take less time to rewrite my code than waiting the 'next step'...

I removed all lease management in my code and nothing has changed...

I strongly believe that it's related to an checkpoint issue..

_host = new EventProcessorHost(Environment.MachineName, eventHubName, consumerGroupName, eventHubConnectionString, checkpointConnectionString);

I use my checkpoint in this method (in my HandledEventProcessor)

    public async Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> events)
    {
        try
        {
            foreach (EventData eventData in events)
            {
                var eventName = eventData.GetEventName();
                var handlers = _handlerFactory.GetHandlers(eventName);
                if (handlers.Any())
                {
                    foreach (var handler in handlers)
                    {
                        SafelyHandleEvent(handler, eventName, eventData, context);
                    }
                }
                else
                {
                    _log.WarnEvent("NoEventHandlers",
                        new Facet { Name = "eventName", Value = eventName });
                }
            }
            await context.CheckpointAsync(); <--- Checkpoint here
        }
        catch (Exception ex)
        {
            _log.ErrorEvent("ProcessEventsFailed", ex,
                new Facet { Name = "eventCount", Value = events.Count() },
                new Facet { Name = "eventHubPath", Value = context.EventHubPath },
                new Facet { Name = "partitionId", Value = context.Lease.PartitionId });
        }
    }

Application Insight logs

23/4/2016 10:35:01 - DEPENDENCY Azure blob: myblobStorage/myContainer Dependency duration: 2.84 ms Successful call: false URL: https://****.blob.core.windows.net:443/myContainer/myConsumerGroupName/partition1

23/4/2016 10:35:01 - DEPENDENCY Azure blob: myblobStorage/myContainer Dependency duration: 2.84 ms Successful call: false URL: https://****.blob.core.windows.net:443/myContainer/myConsumerGroupName/partition0

23/4/2016 10:34:59 - DEPENDENCY Azure blob: myblobStorage/myContainer Dependency duration: 4.4 msSuccessful call: falseURL: https://****.blob.core.windows.net:443/myContainer/myConsumerGroupName/1?comp=lease&timeout=10

23/4/2016 10:34:59 - DEPENDENCY Azure blob: myblobStorage/myContainer Dependency duration: 4.4 msSuccessful call: falseURL: https://****.blob.core.windows.net:443/myContainer/myConsumerGroupName/0?comp=lease&timeout=10

I didn't catch this one... If someone has an idea, it's very welcome...

Any help would be appreciated.

Answer 1

I added a timer befors calling await context.CheckpointAsync();

I think that depending the amount of events, if await context.CheckpointAsync(); is called to many times in short times, this doesn't work...

And of course, I deployed when events are lows..