MassTransit Azure - Receiver from Topic error

Question

I have subscribers to a topic in Azure Service Bus. I am frequently seeing intermittent errors being returned as follows:

MassTransit.Azure.ServiceBus.Core.Transport.ReceiveTransport Error: 0 : ReceiveTransport Faulted: sb://###, System.AggregateException: One or more errors occurred. (One or more errors occurred. (Number must be either non-negative and less than or equal to Int32.MaxValue or -1. Parameter name: dueTime)) ---> System.AggregateException: One or more errors occurred. (Number must be either non-negative and less than or equal to Int32.MaxValue or -1. Parameter name: dueTime) ---> System.ArgumentOutOfRangeException: Number must be either non-negative and less than or equal to Int32.MaxValue or -1. Parameter name: dueTime

I cannot see how/where to set the dueTime in order to prevent this happening. Subscribers are processing messages most of the time.

Is there a way to set dueTime with MassTransit?

Publisher:

await _busControl.Publish(new ConfigurationReloaded(),
                context => context.TimeToLive = new TimeSpan(0, 0, 10, 0), cancellationToken);

Publish Error:

System.AggregateException: One or more errors occurred. (Number must be either non-negative and less than or equal to Int32.MaxValue or -1.
Parameter name: dueTime) ---> System.ArgumentOutOfRangeException: Number must be either non-negative and less than or equal to Int32.MaxValue or -1.
Parameter name: dueTime
   at Microsoft.Azure.ServiceBus.Core.MessageSender.OnSendAsync(IList`1 messageList)
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.Core.MessageSender.SendAsync(IList`1 messageList)
   at MassTransit.Azure.ServiceBus.Core.Transport.ServiceBusSendTransport.SendClientPipe`1.Send(SendEndpointContext clientContext)
   at MassTransit.Azure.ServiceBus.Core.Transport.ServiceBusSendTransport.SendClientPipe`1.Send(SendEndpointContext clientContext)
   at GreenPipes.Agents.PipeContextSupervisor`1.GreenPipes.IPipeContextSource<TContext>.Send(IPipe`1 pipe, CancellationToken cancellationToken)
   at GreenPipes.Agents.PipeContextSupervisor`1.GreenPipes.IPipeContextSource<TContext>.Send(IPipe`1 pipe, CancellationToken cancellationToken)
   at GreenPipes.Agents.PipeContextSupervisor`1.GreenPipes.IPipeContextSource<TContext>.Send(IPipe`1 pipe, CancellationToken cancellationToken)
   at MassTransit.Transports.PublishEndpoint.Publish[T](CancellationToken cancellationToken, T message, PublishEndpointPipeAdapter`1 adapter)
   at MassTransit.Transports.PublishEndpoint.Publish[T](CancellationToken cancellationToken, T message, PublishEndpointPipeAdapter`1 adapter)
   at Services.ConfigurationUpdatedHostedService.StartAsync(CancellationToken cancellationToken) in /src/ConfigurationService/Services/ConfigurationUpdatedHostedService.cs:line 32
   at Microsoft.AspNetCore.Hosting.Internal.HostedServiceExecutor.ExecuteAsync(Func`2 callback)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Hosting.Internal.HostedServiceExecutor.ExecuteAsync(Func`2 callback)
   at Microsoft.AspNetCore.Hosting.Internal.HostedServiceExecutor.StartAsync(CancellationToken token)
---> (Inner Exception #0) System.ArgumentOutOfRangeException: Number must be either non-negative and less than or equal to Int32.MaxValue or -1.
Parameter name: dueTime

Receiver:

 serviceBusHost.ConnectSubscriptionEndpoint<ConfigurationReloaded>($"{_serviceBusOptions.SubscriberName}_{_myUniqueSubscriberName}", x =>
            {
                x.AutoDeleteOnIdle = _serviceBusOptions.TimeToRemoveOnIdle;
                x.Handler<ConfigurationReloaded>(context =>
                {
                    this.Load();
                    return Task.CompletedTask;
                });
            });

Full Stack Trace:

MassTransit.Azure.ServiceBus.Core.Transport.ReceiveTransport Error: 0 : ReceiveTransport Faulted: sb://###, System.AggregateException: One or more errors occurred. (One or more errors occurred. (Number must be either non-negative and less than or equal to Int32.MaxValue or -1.
Parameter name: dueTime)) ---> System.AggregateException: One or more errors occurred. (Number must be either non-negative and less than or equal to Int32.MaxValue or -1.
Parameter name: dueTime) ---> System.ArgumentOutOfRangeException: Number must be either non-negative and less than or equal to Int32.MaxValue or -1.
Parameter name: dueTime
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass64_0.<<ReceiveAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(Int32 maxMessageCount, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.MessageReceivePump.MessagePumpTaskAsync()
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at MassTransit.Azure.ServiceBus.Core.Pipeline.MessageReceiverFilter.GreenPipes.IFilter<MassTransit.Azure.ServiceBus.Core.ClientContext>.Send(ClientContext context, IPipe`1 next)
   at MassTransit.Azure.ServiceBus.Core.Pipeline.MessageReceiverFilter.GreenPipes.IFilter<MassTransit.Azure.ServiceBus.Core.ClientContext>.Send(ClientContext context, IPipe`1 next)
   at GreenPipes.Agents.PipeContextSupervisor`1.GreenPipes.IPipeContextSource<TContext>.Send(IPipe`1 pipe, CancellationToken cancellationToken)
   at GreenPipes.Agents.PipeContextSupervisor`1.GreenPipes.IPipeContextSource<TContext>.Send(IPipe`1 pipe, CancellationToken cancellationToken)
   at GreenPipes.Agents.PipeContextSupervisor`1.GreenPipes.IPipeContextSource<TContext>.Send(IPipe`1 pipe, CancellationToken cancellationToken)
   at MassTransit.Azure.ServiceBus.Core.Transport.ReceiveTransport.<Receiver>b__13_0()
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. (Number must be either non-negative and less than or equal to Int32.MaxValue or -1.
Parameter name: dueTime) ---> System.ArgumentOutOfRangeException: Number must be either non-negative and less than or equal to Int32.MaxValue or -1.
Parameter name: dueTime
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass64_0.<<ReceiveAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(Int32 maxMessageCount, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.MessageReceivePump.MessagePumpTaskAsync()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.ArgumentOutOfRangeException: Number must be either non-negative and less than or equal to Int32.MaxValue or -1.
Parameter name: dueTime
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass64_0.<<ReceiveAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)

Service Collection Extension to add Mass Transit:

 public static IServiceCollection AddMassTransit(
            this IServiceCollection collection, string serviceBusHost, string serviceBusKeyName, string serviceBusSharedAccessKey)
        {
            collection.AddSingleton(Bus.Factory.CreateUsingAzureServiceBus(cfg =>
                {
                    var host = cfg.Host(
                        serviceBusHost,
                        h =>
                        {
                            h.SharedAccessSignature(s =>
                            {
                                s.KeyName = serviceBusKeyName;
                                s.SharedAccessKey = serviceBusSharedAccessKey;
                            });
                        });
                    collection.AddSingleton(host);
                }
            ));

            collection.AddSingleton<IHostedService, BusService>();

            return collection;
        }

Is this happening on the receive transport? I'm not sure why, unless your consumer is scheduling a message (or using redelivery) which your code snippet above doesn't reference. Do you have a call stack associated with the error? — Chris Patterson
@ChrisPatterson Thanks for your assistance. I've added the full stack trace to the question. I haven't configured the topic to use re-delivery or schedule a message (knowingly) — Tolly
dueTime is calculated from TimeToLive, but it should be faulting when you publish. is your Load method throwing an exception, causing the message consumer to fault? — Chris Patterson
I've wrapped the load method and logging out any errors - unfortunately no errors are being thrown so this isn't causing the problem — Tolly
I can see in Azure when I connect a subscriber TimeToLive is being created as 0. I'm not sure why but I'm investigating and I'll post a response if I can reproduce / fix — Tolly

Steve Rash Steve Rash · Accepted Answer · 2019-06-19T18:34:04

I've been suffering from this issue too. After debugging I think I found the issue. There is a default value in the .net core service bus library that is too low.

In the ShareAccessSignature config method setting TokenTimeToLive to a reasonable value seemed to solve.

                        sbcfg.SharedAccessSignature(sas =>
                    {
                        sas.TokenTimeToLive = TimeSpan.FromMinutes(20);
                        sas.KeyName = Configuration.GetValue<string>("ServiceBus:KeyName");
                        sas.SharedAccessKey = Configuration.GetValue<string>("ServiceBus:AccessKey");
                    });

For info these are the related (open) issues in github related to this default behaviour.

https://github.com/Azure/azure-service-bus-dotnet/issues/672

https://github.com/Azure/azure-sdk-for-net/issues/6312

MassTransit Azure - Receiver from Topic error

1 Answers