Rebus Retry policy

Question

I have questions on Rebus retry policy below:

Configure.With(...)
    .Options(b => b.SimpleRetryStrategy(maxDeliveryAttempts: 2))
    .(...)

https://github.com/rebus-org/Rebus/wiki/Automatic-retries-and-error-handling#customizing-retries

1 Can it be used for both Publiser (enqueue messages) and Subscriber (dequeue messages)?

2 I have a subscriber that is unable to dequeue the message. Thus the message is sent to error queue.

Below is error for when putting the message to error queue. But I cannot see the loggings for the retry.

[ERR] Rebus.Retry.PoisonQueues.PoisonQueueErrorHandler (Thread #9): Moving messa
ge with ID "<unknown>" to error queue "poison"
Rebus.Exceptions.RebusApplicationException: Received message with empty or absen
t 'rbs2-msg-id' header! All messages must be supplied with an ID . If no ID is p
resent, the message cannot be tracked between delivery attempts, and other stuff
 would also be much harder to do - therefore, it is a requirement that messages
be supplied with an ID.

Is it possible to define and store custom logging for each retry, not within IErrorHandler?

3 How long does each retry wait in between be default?

4 Is it possible to define custom wait time for each retry (not within IErrorHandler)? If so, is Polly supported for this scanario? like below:

Random jitterer = new Random(); 
Policy
  .Handle<HttpResponseException>() // etc
  .WaitAndRetry(5,    // exponential back-off plus some jitter
      retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))  
                    + TimeSpan.FromMilliseconds(jitterer.Next(0, 100)) 
  );

https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/implement-resilient-applications/implement-http-call-retries-exponential-backoff-polly

Update

How can I test the retry policy?

Below is what I tried based on the code below:

public class StringMessageHandler : IHandleMessages<String>
{   
    public async Task Handle(String message) 
    {
          //retry logic with Polly here
    }   
}

I sent an invalid message of string type to the string topic, however, the Handle(String message) is not invoked at all.

What do you mean by (...) I sent an invalid message of string type to the string topic (...)? Did you publish it? Did you remember to subscribe to it? — mookid8000
I sent an invalid message using Azure Service Bus Explorer. The message only has a body, no header. I didn't publish it via Rebus. This is only to test the retry logic, but the Handle() is not called. However, the invalid message is moved to error queue by Rebus. Is it possible to hook into the retry logic so that logging and Polly policy can be performed? My understanding of Handle method might be different from Rebus. — Pingpong
Your message gets immediately moved to the dead-letter queue because it doesn't have a rbs2-message-id header – that header is the bare minimum required for Rebus to even try to handle a message, as it's necessary for the retry mechanism to work. — mookid8000
Sorry for not making it clearer. My main question is how to test the custom retry logic via Polly within StringMessageHandler.Handle(String message) method? — Pingpong
You class is just a class that implements an interface – it doesn't really contain any Rebus-related logic, so I suggest you just call the Handle method and verify that your Polly policy works :) — mookid8000

mookid8000 mookid8000 · Accepted Answer · 2019-05-16T11:59:54

Rebus' retry mechanism is only relevant when receiving messages. It works by creating a "queue transaction"(*), and then if message handling fails, the message gets rolled back to the queue.

Pretty much immediately thereafter, the message will again be received and attempted to be handled. This means that there's no delay between delivery attempts.

For each failed delivery, a counter gets increased for that message's ID. That's why the message ID is necessary for Rebus to work, which also explains why your message with out an ID gets immediately moved to the dead-letter queue.

Because of the disconnected nature of the delivery attempts (only a counter per message ID is stored), there's no good place to hook in a retry library like Polly.

If you want to Pollify your message handling, I suggest you carry out individual operations with Polly policies – that way, you can easily have different policies for dealing with failing web requests, failing file transfers on network drives, etc. I do that a lot myself.

To avoid not being able to properly shut down your bus instance if it's in the process of a very long Polly retry, you can pass Rebus' internal CancellationToken to your Polly executions like this:

public class PollyMessageHandler : IHandleMessages<SomeMessage>
{
    static readonly IAsyncPolicy RetryPolicy = Policy
        .Handle<Exception>()
        .WaitAndRetryForeverAsync(_ => TimeSpan.FromSeconds(10));

    readonly IMessageContext _messageContext;

    public PollyMessageHandler(IMessageContext messageContext)
    {
        _messageContext = messageContext;
    }

    public async Task Handle(SomeMessage message) 
    {
        var cancellationToken = _messageContext.GetCancellationToken();

        await RetryPolicy.ExecuteAsync(DoStuffThatCanFail, cancellationToken);
    }

    async Task DoStuffThatCanFail(CancellationToken cancellationToken)
    {
        // do your risky stuff in here
    }
}

(*) The actual type of transaction depends on what is supported by the transport.

With MSMQ, it's a MessageQueueTransaction object that has Commit() and Rollback() methods on it.

With RabbitMQ, Azure Service Bus, and others, it's a lease-based protocol, where a message becomes invisible for some time, and then if the message is ACKed within that time, then the message is deleted. Otherwise – either if the message or NACKed, or if the lease expires – the message pops up again, and can again be received by other consumers.

With the SQL transports, it's just a database transaction.

Rebus Retry policy

1 Answers