AWS SQS BackUp Solution Design

2
votes

Problem Statement

Informal State

We have some scenarios where the integration layer (a combination of AWS SNS/SQS components and etc.) is also responsible for the data distribution to target systems. Those are mostly async flows. In this case, we send a confirmation to a caller that we have received the data and will take a responsibility for the data delivery. Here, although the data is not originated from the integration layer we are still holding it and need to make sure that the data is not lost, for example, if the consumers are down or if messages, on-error, are sent to the DLQs and hence being automatically deleted after the retention period.

Solution Design

Currently my idea was to proceed with a back-up of the SQS/DLQ queues based upon CloudWatch configured alerts using ApproximateAgeOfOldestMessage metric with some applied threshold (something like the below):

Msg Expiration Event if ApproximateAgeOfOldestMessage / Message retention > Threshold

Now, more I go forward with this idea and more I doubt that this might be actually the right approach…

In particular, I would like to build something unobtrusive that can be "attached" to our SQS queues and dump the messages that are about to expire in some repository, like for example the AWS S3. Then have a procedure to recover the messages from S3 to the same original queue.

The above procedure contains many challenges like: message identification and consumption (receive message is not design to "query" for specific messages), message dump in the repository with a reference to the source queue, etc. which would suggest to me that the above approach might be a complex over-kill.

That being said, I'm aware of other "alternatives" (such as this) but I would appreciate if you could answer to the specific technical details described above, without trying to challenge the "need" instead.

1
amazon-web-servicesamazon-sqsbackup-strategies
Do you have the ability to change the code that inserts into the queue, and the code that deletes messages from the queue? If so I would also insert a copy of the message in a DynamoDB table at the time you add it to the queue, and delete the record from the DynamoDB table when you delete it from the queue. Then all messages that timed out and were deleted from SQS would still be in the DynamoDB table and you could query the table for records older than 14 days to find them. - Mark B
Hi @MarkB, thanks for the answer. Yes, I have that ability or option possibility, indeed I have full control on the component/s which write in the SQS queues. This indeed is the "alternative" approach which is similar to the one suggested in the link I mentioned above though your suggested approach is something slightly different and I like it. The only concern that I have with this approach, is that it is a little more intrusive since the code that is handling business logic, should now take care also about, another, orthogonal concern. - Indrit
I understand, but I'm not aware of any other way to detect that an SQS message has been deleted due to the 14 day expiration. An expired message isn't moved into the DLQ is it? It's just deleted I think. So something has to happen at the time your code deletes the message from the queue to keep your tracking accurate. You could create an API that the business logic calls instead of deleting the SQS message directly. The API it calls could handle both steps, deleting the message from the queue as well as the DynamoDB table. - Mark B
Hi @MarkB, yes, I could build a sort of a wrapper component that could encapsulate the double-call logic, still I need to place this component at the front of the communication with queue, intercepting 'delete' message. I'm afraid this would put this new, custom build, component in the critical path as far as reliability is concerned...which I would like to avoid. Yes, an expired msg is just deleted and if I'm not wrong, the age is cumulative when moved into DLQ. Not sure if message-deletion(or expiration) itself could be intercepted, until now I haven't found any evidence though. - Indrit
Moreover what happens if the delete from DynamoDB fails? It will inject further complexity in the system. - Indrit

1 Answers

0
votes

Similar to Mark B's suggestion, you can use the SQS extended client (https://github.com/awslabs/amazon-sqs-java-extended-client-lib) to send all your messages through S3 (which is a configuration knob: https://github.com/awslabs/amazon-sqs-java-extended-client-lib/blob/master/src/main/java/com/amazon/sqs/javamessaging/ExtendedClientConfiguration.java#L189).

The extended client is a drop-in replacement for the AmazonSQS interface so it minimizes the intrusion on business logic - usually it's a matter of just changing your dependency injection.