Ensure at-most-once semantic with SendGrid Mail API

Question

I have an [Azure Storage] queue where I put e-mail messages to be sent. Then, there is a separate service which monitors that queue and send e-mails using some service. In this particular case I'm using SendGrid.

So, theoretically, if the sender crashes right after a successful call to SendGrid Mail Send API (https://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/index.html), the message will be returned to the queue and retried later. This may result in the same e-mail being delivered more than once, which could be really annoying for some type of e-mail.

The normal way to avoid this situation would be to provide some sort of idempotency key to Send API. Then the side being called can make sure the operation is performed at most once.

After careful reading of SendGrid documentation and googling, I could not find any way to achieve what I'm looking for here (at most once semantic). Any ideas?

Answer 1

Without support for an idempotency key in the API itself your options are limited I think.

You could modify your email sending service to dequeue and commit before calling the Send API. That way if the service fails to send the message will not be retried as it has already been removed from the queue, it will be sent at most once.

Additionally, you could implement some limited retries on particular http responses (e.g. 429 & 5xx) from SendGrid where you are certain the message was not sent and retrying might be useful - this would maintain "at most once" whilst lowering the failure rate. Probably this should include some backoff time between each attempt.