using RabbitMQ to sync up remote server transactions

1
votes

I have a Rails app installed on several remote servers. The Rails app generates financial transactions and I'd like all the remote servers send this metadata to a central hub where they are stored in a relational DB. There MIGHT be other processing tasks on the transactions. These servers are not under my control so I can't guarantee constant connection to the internet. They will be connected to the internet at least a couple of hours each day.

Is RabbitMQ useful for such an architecture? The Exchange and the Queues would be installed on the remote servers (along with the Producer Rails app), right? Would I have one consumer service per each remote server, or could I have one central consumer service for all the remote servers/queues?

Bonus questions since I'm such a noob in the world of RabbitMQ:

  1. what port would I need to open on the remote machines such that my consumer can consume the data from the queue?
  2. I would use a Direct Exchange, right?
  3. how many messages can accumulate in the queue before being consumed? Is there a chance that things might crap out if the remote server don't get connected to the internet for a couple of days (worst scenario)?

RabbitMQ architecture

1
rabbitmq

1 Answers

1
votes

Due to the heavily disconnected nature of the system (which isn't the typical use of an AMQP system, I don't think), and the fact that you need to push only a few times a day perhaps, I wonder if you wouldn't be better to skip with whole AMQP/RabbitMQ thing and instead write direct to a local database with a scheduled task to run a nightly push of that data to the central server instead.

Keep in mind that RabbitMQ is a message brokerage service with the inherent ability to store messages - not a message storage service. The primary intent of the service is to disconnect the messages from their publishing system and push them where they need to go, typically in a realtime manner. A database, on the other hand, is of course a storage system, with plenty of options for pushing data around in a non-realtime manner.

With that said, here's some thoughts regarding RabbitMQ in this context:

First, some context would help: we have no idea how many transactions a day your rails servers are handling and wanting to push. 1 per day? 1 million per day?

Regarding backlogged messages in queues: RabbitMQ is (apparently) quite scalable, and I would think this shouldn't be an issue in regards to the message service itself (here's some info from their site on how queues are managed: http://www.rabbitmq.com/blog/2011/10/27/performance-of-queues-when-less-is-more/ )

The available memory and hard drive space (when configured correctly) should be the limiting factors on how many messages you can load into the queue.

https://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/

https://www.rabbitmq.com/maxlength.html

https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine

Regarding the consumer setup, I don't see any reason you couldn't have a central service connecting to the various RabbitMQ servers. You'll need to run separate connections and channels of course (and you may wish to run a few instances of this to help round-robin the incoming messages).

Regarding the exchange type: Direct seems appropriate for this, yes.