ingesting in azure eventhub fails every morning

Question

sending events to eventhub fails every early morning with the below message. I have masked the number, since I am not sure what is that number and for security reasons.(code and error below)

I consistently send ~240 events per minute of very small size, so that should not be the problem.
I have 2 partitions and 2 consumers for my eventhub Please let me know if any of you know the solution or need more info. Thanks!

Error: Message :An error occurred during communication with 'XXXXX06f186e4bb89aea2d8959bXXXXX_GXX'. Check the connection information, then retry.

StackTrace : at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Common.AsyncResult1.End(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.Amqp.FaultTolerantObject1.OnEndCreateInstance(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.SingletonManager1.EndGetInstance(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessageSender.OnEndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.ClientEntity.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.OpenOnceManager.OnEndCreateInstance(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.SingletonManager1.EndGetInstance(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.OpenOnceManager.OpenOnceManagerAsyncResult1.OpenComplete(IAsyncResult result) at Microsoft.ServiceBus.Common.AsyncResult.SyncContinue(IAsyncResult result) at Microsoft.ServiceBus.Messaging.OpenOnceManager.OpenOnceManagerAsyncResult1..ctor(OpenOnceManager openOnceManager, TimeSpan openTimeout, AsyncCallback callback, Object state, Func3 beginOperation, EndOperation1 endOperation) at Microsoft.ServiceBus.Messaging.OpenOnceManager.Begin(AsyncCallback callback, Object state, Func3 beginOperation, Action1 endOperation) at Microsoft.ServiceBus.Messaging.MessageSender.BeginSendEventData(TrackingContext trackingContext, IEnumerable1 eventDatas, TimeSpan timeout, AsyncCallback callback, Object state) at Microsoft.ServiceBus.Messaging.EventHubSender.<>c__DisplayClass23_0.b__0(AsyncCallback c, Object s) at System.Threading.Tasks.TaskFactory1.FromAsyncImpl(Func3 beginMethod, Func2 endFunction, Action1 endAction, Object state, TaskCreationOptions creationOptions) at Microsoft.ServiceBus.Common.Parallel.TaskHelpers.CreateTask(Func3 begin, Action`1 end, Object state) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() Date :4/12/2017 2:48:17 AM

Code: EventHubClient deveventHubClient = EventHubClient.CreateFromConnectionString(connectionString, "EventHubName");

private async Task SendInBatch(IList<byte[]> source, Guid UUID)
{
List events = new List(source.Select(b => new EventData(b)));

    EventHubSender Partition0;
    EventHubSender Partition1;
    Partition0 = deveventHubClient.CreatePartitionedSender("0");
    Partition1 = deveventHubClient.CreatePartitionedSender("1");
    if (UUID.GetHashCode() % 2 == 0)
    {
        await Partition0.SendBatchAsync(events);
    }
    else
    {
        await Partition1.SendBatchAsync(events);
    }
    return Unit.Default;
}

Does this issue appear every morning? Do you try to use another event hub to test if same issue appear? — Fei Han
I tried 2 eventhubs, overnight 1 failed at ~2:40 AM and another at ~4:30 AM., and this happens consistently every day. — Radhika Bhala

Sreeram Garlapati Sreeram Garlapati · Accepted Answer · 2017-04-13T16:50:11

Short ANS:

You are possibly experiencing transient error while sending. If you want absolutely highly available code - consider changing your code to EventHubClient.sendBatchAsync(...) instead of using PartitionedSender.sendBatchAsync(...) and get rid of % 2 logic in your code.

WHY:

Imagine - EventHubs as a stream of Events on cloud/azure.

In a distributed environment - faults are bound to happen all the time, although probability is low - for ex: the VM hosting your EventHubs could restart, one of n/w gateways could fail and the connections might reset, the container hosting your code might experience high-load and our load balancing algo. might move it around etc.,

When faults happen, we want to enable our customers to be able to send to EventHubs without perceiving the fault. So, in-other-words, to offer high-availability to our customers, we exposed the notion of partitions.

When you are sending using EventHubClient.sendBatchAsync(...) - our Gateway will detect which eventhub partition is immediately available and will route the event to that Specific event hubs partition - if multiple partitions are available - it will round-robin (for ex: just like how you did using %2).

Simply put, no. of partitions of EventHubs determines the degree of High-availability of EventHubs. So, if you are sensitive to connection failures - consider moving out of creating a PartitionedSender and use EventHubClient to directly send events to EventHub.

ingesting in azure eventhub fails every morning

1 Answers