Amazon SQS - polling multiple queues on multiple threads

Question

I'm setting up bounce, complaint, and delivery notifications for Amazon SES using their SQS Queue Service.

The documentation recommends that when polling multiple SQS queues, to do it on multiple threads. This is especially true when using long-polling since long-polling multiple queues on one thread could cause the queue being polled to delay new messages from different queues.

So if I want to poll 3 SQS queues in my ASP.NET MVC application, should I just spawn barebones threads like this:

 Thread bouncesThread = new Thread(() => 
 {
      IAmazonSQS sqsClient = AWSClientFactory.CreateAmazonSQSClient(RegionEndpoint.USWest2);
      while(true) 
      {
          //this will long-poll
          ReceiveMessageResponse receiveMessageResponse = sqsClient.ReceiveMessage(bouncesQueueRequest);

          ProcessQueuedBounce(receiveMessageResponse);
      }
 });

 bouncesThread.SetApartmentState(ApartmentState.STA);
 bouncesThread.Start();

Or is there a better way to do this, since this is a long running process and should run for the length of the application? Should I use IRegisteredObject?

Whats the recommended way to do this?

Krease Krease · Accepted Answer · 2014-10-14T22:58:38

The "recommended" way to do this will vary based on your use case and what your bottlenecks for processing the messages are. Splitting up into threads based on the queue is a great start to prevent SQS_QueueA from being blocked by an empty SQS_QueueB.

You may also want to:

Add more threads for a specific queue. For example, if you have a particularly busy sqs queue, you may have multiple threads receiving messages from it. This will be notably beneficial if you have higher latency to SQS.
Split receiving messages from processing messages - have a separate threadsafe queue for holding ReceiveMessageResponse objects, then have a thread (or group of threads) specifically responsible for processing results from that queue.

Combining the above, you can scale your SQS polling and result processing separately depending on which is the bottleneck.

If your threads are doing very similar things (in this case, polling an SQS queue and processing a message), you can probably share some or all of the code between those threads, making the differences just as parameters to be passed in.

Amazon SQS - polling multiple queues on multiple threads

1 Answers