Timeout manager retuning timout immediately

Question

I have a very simple scheduler saga, that is supposed to send a specific message every day. It is implemented as a saga that requests a timeout. when the timeout is handled an action is executed (a message is sent) and a new timeout is requested the next day.

I have done the exact same thing before with success, but now the timeout seems to trigger immediately so matter what DateTime is requested.

The endpoint is self-hosting, and configured to use InMemoryPersistence. NServiceBus version is 6.4.3.

The saga is implemented as follows. I have removed all the logic, but still timeout messages are received immediately and infinitely.

public class SchedulerSaga: Saga<SchedulerState>,
    IAmStartedByMessages<StartSchedulerSagaCommand>,
    IHandleTimeouts<SchedulerTimeout>
{
    private readonly IConfigurationProvider _config;

    public SchedulerSaga(IConfigurationProvider config)
    {
        _config = config;
    }

    protected override void ConfigureHowToFindSaga(SagaPropertyMapper<SchedulerState> mapper)
    {
        mapper.ConfigureMapping<StartSchedulerSagaCommand>(_ => _.SchedulerName).ToSaga(_ => _.SchedulerName);
    }

    public async Task Handle(StartSchedulerSagaCommand message, IMessageHandlerContext context)
    {
        Data.SchedulerName = message.SchedulerName;
        await StartProcessAndScheduleNewTimeout(context);
    }

    public async Task Timeout(SchedulerTimeout state, IMessageHandlerContext context)
    {
        Data.Counter++;
        await StartProcessAndScheduleNewTimeout(context);
    }

    private async Task StartProcessAndScheduleNewTimeout(IMessageHandlerContext context)
    {
        await RequestTimeout(context, new DateTime(2018, 9, 16, 0, 0, 0, DateTimeKind.Utc), new SchedulerTimeout { Counter = Data.Counter });
    }
}

The endpoint is configured as follows:

    public static EndpointConfiguration ConfigureMsmqEndpoint(IWindsorContainer container)
    {
        var endpointConfiguration = new EndpointConfiguration(MsmqEndpointName);

        ConfigureRouting(endpointConfiguration);

        endpointConfiguration.UsePersistence<InMemoryPersistence>();

        endpointConfiguration.SendFailedMessagesTo($"{MsmqEndpointName}.error");
        endpointConfiguration.AssemblyScanner().ExcludeAssemblies("tools");
        endpointConfiguration.EnableInstallers();

        ConfigureUnobtrusiveMessageConvention(endpointConfiguration);

        endpointConfiguration.Recoverability().Delayed(DelayedSettings);

        endpointConfiguration.UseContainer<WindsorBuilder>(c => c.ExistingContainer(container));

        return endpointConfiguration;
    }

I also tried to use the built in Scheduling mechanism and the same thing happens, hundereds of timeouts are triggered every second.

await endpointInstance.ScheduleEvery(
        timeSpan: TimeSpan.FromMinutes(5),
        task: context=> context.SendLocal(new SomeMessage())
    )
.ConfigureAwait(false);

Update: Add repo with code reproducing the problem.

https://github.com/spinakr/nsb-scheduling-msmq

The problem only occur when the package "NServiceBus.Azure.Transports.WindowsAzureStorageQueues" is references in the project, even if it is not used!

The app in review has two endpoint hosted in the same process. Consuming messages from MSMQ and Azure storage queues. In the repo, it is only when adding the azure storage queues transport package that the issue start occurring.

Could you confirm that a single starting message is sent and incoming queue doesn't contain previously sent messages? — Sean Feldman
Yes, definitely. I tried debugging, and breaking in both the message handler and timeout handler, only timeouts are received, this can also be observed in the journal of the msmq queue. — sp1nakr
Also take note that the same behaviour is happening when using the endpointInstance.ScheduleEvery() function. This leads my to believe that the problem is isolated to the timeout manager, as both ScheduleEvery and RequestTimeout uses it. — sp1nakr
Have tried it with NServiceBus 6.4.3 and it works as expected (timeout is sent and doesn't kick in for a large timeout). If modified for a shorter period of time, timeout is delivered as expected. Could you share the code you used to reproduce this issue as a standalone project on GitHub or similar? — Sean Feldman
Thanks for the answer @SeanFeldman. I will test it out in isolation when I get back to work. I guess it must be something with the configuration or my environment. — sp1nakr

KatoStoelen KatoStoelen · Accepted Answer · 2018-01-30T11:22:40

I assume that the endpoint you are describing are using MSMQ as transport (based on names of methods and fields) and that the saga is running on that endpoint.

MSMQ relies on the Timeout Manager to support delayed delivery. The Azure Storage Queues transport, on the other hand, seems to solve delayed delivery in a different way. It actually has a feature, enabled by default, that prevents deferred messages from being routed to the Timeout Manager. If your MSMQ endpoint were to scan the NServiceBus.Azure.Transports.WindowsAzureStorageQueues assembly, your deferred messages would not reach the Timeout Manager.

One solution to this issue would be to configure the assembly scanner in your MSMQ endpoint to exclude that assembly:

endpointConfiguration.AssemblyScanner().ExcludeAssemblies(
    "NServiceBus.Azure.Transports.WindowsAzureStorageQueues.dll");

Timeout manager retuning timout immediately

2 Answers