Using S3 to group SQS messages instead of FIFO queues

0
votes

We plan to use SQS standard queue for messaging. The messages are a bunch of records that need to be received in order at the consumer end. There are a few reasons due to which we plan not to use FIFO queues.

  1. FIFO queues have a few restrictions as noted here and are recommended only for few selected use cases.
  2. We have multiple producers that push such messages to the queue (all producers are independent of the other) hence we are most likely to hit the 300 messages per second limitation.

Given this we are evaluating the SQS extended library support to use S3 for storing message payloads. We would club all the linked records into one message and post it to SQS as one request. I have a few questions

  1. What are the limitations or side effects of using S3 for persisting message payloads? One that I am aware off would be - S3 cost - I am assuming this won't be large given our messages don't cross a few MB's max.
  2. Are there real world examples of using this approach over FIFO queues for grouping messages?
1
amazon-s3amazon-sqsfifo
I don't understand how using S3 for message payloads replaces FIFO functionality - they are not related. If your messages are small (and will fit in SQS message size), consider plain SQS. Another alternative to consider for your use case is AWS Kinesis, though it's hard to tell which is better for you without more details – Krease
I am referring to SQS + S3 for messaging. Since I want to transfer a bunch of records in ordered fashion which won't fit in the 256 KB SQS limit, I was considering extended S3 support. FIFO also provides ordered delivery but it needs to be evaluated thoroughly. Hence the question. – Andy Dufresne

1 Answers

1
votes

S3 introduces additional latency (at each end of the queue), depending on the payload size, whether the message publishers/consumers are in AWS or hosted elsewhere, and how much bandwidth an individual server instance has available to it. (Off the cuff, I’d guess it’s >200 ms for a 1 MB payload.) The cost will be pretty insignificant, especially if you set up appropriate bucket lifecycle policies to archive or delete old data. Don’t forget that S3 is strongly consistent on the initial create, but only eventually consistent for any updates to an object. If possible, don’t update an object once you have created it.

I don’t have any real world examples, but I’ll let you know if I find one.

You will probably find it simpler to implement what you need using some sort of database, as was suggested in the article that you linked (which explained the limitations of FIFO queues). Make sure your decision isn’t biased by looking for premature optimizations.