API design around RabbitMQ for publisher/subscriber

Question

TL;DR - Whats the best way to expose RabbitMQ to a consumer via REST API?

I'm creating an API to publish and consume message from RabbitMQ. In my current design, the publisher is going to make a POST request. My API will route the POST request to the exchange. In this way, the publisher doesn't have to know the server address, exchange name etc. while publishing.

Now the consumer part is where I'm not sure how to proceed.

At the beginning there will be no queues. When a new consumer wants to subscribe to a TOPIC, then I will create a queue and bind it to the exchange. I need help with answers to few questions -

Once I create a queue for the consumer, what's the next step to let the consumer get messages from that queue?
I make the consumer ask for a batch of messages(say 50 messages) from the queue. Then once I receive an ack from the consumer I will send the next 50 messages from queue. If I don't receive an ack I will requeue the 50 messages back into the queue. Isn't this expensive in terms of opening and closing connection between the consumer and my API?

If there is a better approach then please suggest

Ugh, hard to answer without knowing a real need. Pls, add you target language so that you can get a better recommendation. In short: 1) to consume you need to start consuming, on AMQP protocol level it is a basic.consume method, in client libraries regardless language it should be called in a similar way. 2) You basically not sending any messages from queue to consumer, it's broker (RabbitMQ) responsibility. You send messages to exchanges, bind queues to exchanges and consume from queues. But you can interact between consumer and sender if it fits your needs. Real costs should be measured. — pinepain

Derick Bailey Derick Bailey · Accepted Answer · 2016-08-04T13:34:50

In general, your idea of putting RMQ behind a REST API is a good one. You don't want to expose RMQ to the world, directly.

For the specific questions:

Once I create a queue for the consumer, what's the next step to let the consumer get messages from that queue?

Have you read the tutorials? I would start there, for the language you are working with: http://www.rabbitmq.com/getstarted.html

Isn't this expensive in terms of opening and closing connection between the consumer and my API?

Don't open and close connections for each batch of messages.

Your application instance (the "consumer" app) should have a single connection. That connection stays open as long as you need it - across as many calls to RabbitMQ as you want.

I typically open my RMQ connection as soon as the app starts, and I leave it open until the app shuts down.

Within the consumer app, using that one single connection, you will create multiple channels through the connection. A channel is where the actual work is done.

Depending on your language, you will have a single channel per thread; a single channel per queue being consumed; etc

You can create and destroy channels very quickly, unlike connections.

More specifically with your idea of batch processing, this will be handled by putting a consumer prefetch limit on your consumer and then requiring messages to be acknowledged after processing it.

API design around RabbitMQ for publisher/subscriber

1 Answers