Why to use separate polling and worker threads?

Question

Context: I am designing an application which will be consuming messages from various Amazon SQS queues. (More than 25 queues) For this, I am thinking of creating a library to consume messages from the queues, (call it MessageConsumer)

I want to be dynamically allocating threads to receive/process messages from different queues based on traffic in the queue to minimise waste of resources. There are 2 ways I can go about it.

1) Can have only one type of thread that polls queues, receives messages and process those message and have one common thread pool for all queues. 2) Can have separate polling and worker threads.

In the second case, I will be having common worker thread pool and constant number of pollers per queue.

Edit: To elaborate on the second case: I am planning to have 1 continuously running thread per queue to poke that queue for the amount of messages in it. Then have some logic to decide the number of polling threads required per queue based on the number of messages in each queue and priority of the queue.

I dont want polling threads running all the time because that may cause empty receives (sqs.receiveMessages()), so I will allocate the polling threads based on traffic. The high traffic queues will have more polling threads and hence more jobs being submitted to worker thread pool.

Please suggest any improvements or flaws in this design?

What's the point in separating pollers and workers? What do the pollers do? They poll items from the queue and then what? They put them into a (local) queue for workers to pick them up? — Fildor
@Fildor Thanks for the response. Yes, in my case, pollers will receive messages and then submit a job to worker to process that message (maybe deserialize them or handle them in some specific way). I have updated the question with more details. Would be really very helpful if u have some more feedback — learning_to_write_good_code
It's hard to tell from the description, but what you're describing sounds a bit like using an AWS Lambsa with SQS integration, but you'd need to research yourself whether it fits your use-case — matt freake

John Rotenstein John Rotenstein · Accepted Answer · 2020-04-14T08:59:32

The recommended process is:

Workers poll the queue using Long Polling (which means it will wait for a maximum of 20 seconds before returning an empty response)
They can request up to 10 messages per call to ReceiveMessage()
The worker processes the message(s)
The worker deletes the message from the queue
Repeat

If you wish to scale the number of workers, you can base this on the ApproximateNumberOfMessagesVisible metric in Amazon CloudWatch. If the number goes too high, add a worker. If it drops to zero (or below some threshold), remove a worker.

It is probably easiest to have each worker only poll one queue.

There is no need for "pollers". The workers do the polling themselves. This way, you can scale the workers independently, without needing some central "polling" service trying to manage it all. Simply launch a new Amazon EC2 instance, launch the some workers and they start processing messages. When scaling-in, just terminate the workers or even the instance -- again, no need to register/deregister workers with a central "polling" service.

Why to use separate polling and worker threads?

1 Answers