Azure EventHub sender fails to send messages

Question

I use latest WindowsAzure.ServiceBus nuget package and simple code snippet to send messages to Azure EventHub from my Cloud Service WebRole api controller:

var eventData = new EventData(buffer) { PartitionKey = partition };
await _eventHubClient.SendAsync(eventData);

After a while, sender fails to send messages and I see following exceptions in logs every time I try to send a message:

Microsoft.ServiceBus.Messaging.MessagingCommunicationException: 
An error occurred during communication with 'N/A'. Check the connection information, then retry. 
---> System.InvalidOperationException: Failed with error code 0xe. 
Server stack trace: at Microsoft.ServiceBus.Tracing.EventSource..ctor(Boolean disableTracing) 
   at Microsoft.ServiceBus.Tracing.MessagingClientEtwProvider.get_Provider() 
   at Microsoft.ServiceBus.Messaging.Amqp.Sasl.SaslTransportProvider.AddHandler(SaslHandler handler) 
   at Microsoft.ServiceBus.Messaging.Amqp.AmqpTransportSettings.CreateAmqpSettings(String sslHostName, Int32 maxFrameSize, Boolean forceTokenProvider) 
   at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessagingFactory.ConnectAsyncResult.<GetAsyncSteps>d__13.MoveNext() 
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.Start() Exception rethrown at [0]: 
   at Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception exception) at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) 
   at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessagingFactory.ConnectAsyncResult.End(IAsyncResult result, ConnectInfo& info) 
   at Microsoft.ServiceBus.Messaging.Amqp.FaultTolerantObject`1.CreateAsyncResult.<>c.<GetAsyncSteps>b__5_1(CreateAsyncResult thisPtr, IAsyncResult r) 
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state) 
Exception rethrown at [1]: 
   at Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception exception) at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) 
   at Microsoft.ServiceBus.Messaging.Amqp.FaultTolerantObject`1.OnEndCreateInstance(IAsyncResult asyncResult) 
   at Microsoft.ServiceBus.Messaging.SingletonManager`1.EndGetInstance(IAsyncResult asyncResult) 
   at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessagingFactory.OpenLinkAsyncResult.<>c.<GetAsyncSteps>b__35_3(OpenLinkAsyncResult thisPtr, IAsyncResult r) 
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state) 
--- End of inner exception stack trace --- 
   at Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception exception) at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) 
   at Microsoft.ServiceBus.Messaging.Amqp.FaultTolerantObject`1.OnEndCreateInstance(IAsyncResult asyncResult) 
   at Microsoft.ServiceBus.Messaging.SingletonManager`1.EndGetInstance(IAsyncResult asyncResult) 
   at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessageSender.OnEndOpen(IAsyncResult result) 
   at Microsoft.ServiceBus.Messaging.ClientEntity.EndOpen(IAsyncResult result) 
   at Microsoft.ServiceBus.Messaging.OpenOnceManager.OnEndCreateInstance(IAsyncResult asyncResult) 
   at Microsoft.ServiceBus.Messaging.SingletonManager`1.EndGetInstance(IAsyncResult asyncResult) 
   at Microsoft.ServiceBus.Messaging.OpenOnceManager.OpenOnceManagerAsyncResult`1.OpenComplete(IAsyncResult result) 
   at Microsoft.ServiceBus.Messaging.OpenOnceManager.OpenOnceManagerAsyncResult`1..ctor(OpenOnceManager openOnceManager, TimeSpan openTimeout, AsyncCallback callback, Object state, Func`3 beginOperation, EndOperation`1 endOperation) 
   at Microsoft.ServiceBus.Messaging.OpenOnceManager.Begin(AsyncCallback callback, Object state, Func`3 beginOperation, Action`1 endOperation) 
   at Microsoft.ServiceBus.Messaging.MessageSender.BeginSendEventData(TrackingContext trackingContext, IEnumerable`1 eventDatas, TimeSpan timeout, AsyncCallback callback, Object state) at Microsoft.ServiceBus.Messaging.EventHubClient.<>c__DisplayClass32_0.<SendAsync>b__0(AsyncCallback c, Object s) 
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions) 
   at Microsoft.ServiceBus.Common.Parallel.TaskHelpers.CreateTask(Func`3 begin, Action`1 end, Object state) 
--- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Seems once sender stumble upon this error it can't send messages anymore.

@PeterBons It looks more like a problem with creating EventSource when trying to add Etw event and nothing to do with a connection or message sending — Max Brodin

Serkant Karaca Serkant Karaca · Accepted Answer · 2016-11-08T22:25:08

Error code 0xE is ERROR_OUTOFMEMORY from tracing. As a workaround you can either scale up the web role instance or disable tracing completely.

I have filed a bug to fix this on our client as well. Basically, messaging operations should not fail due to tracing issues.

Azure EventHub sender fails to send messages

1 Answers