Amazon SQS queue: getting a response from queue

Question

Image processing system.

User uploads a photo to the S3. Image processing application takes the photo from S3 and starts processing on it and returns an output.

Process: Webserver is sending SQS message with info of image to the queue, and the Image processing Server gets the SQS message and starts processing over this image. As soon as the image processing is finished it uploads the image back to S3 and returns the SQS message back to queue. Image Processing engine is on the EC2 server.

Here is the question: How can the webserver get the return SQS message from queue which was sent from Image processing server?

Answer 1

Don't think of image processing as a remote procedure call, where there's a request and a response -- think of it as a series of tasks being performed by the different components in your distributed system. Solve this problem with two queues: uploaded-images and processed-images.

A web server accepts the image, puts it in S3, and posts a message to the uploaded-images queue indicating that it needs to be processed.

Your image processing EC2 instances wait for messages to arrive on the uploaded-images queue, then retrieve the image from S3, do some processing, and put the results in S3. They then send a message to the processed-images queue indicating that the image has been processed.

Your web servers poll processed-images, and do whatever they need to do once an image is processed: update a database, set an entry in a short-lived cache indicating the image is done processing, etc.