Masstransit rabbitMQ publishing on many threads

Question

I have an application that spins up a lot threads. Each thread submits the same message type via masstransit / rabbitmq. I've created a singleton to store one instance of IBus. The application only publishes messages so no endpoints are in the configuration.

Is a singleton message bus the correct thing to do when only publishing? I did performance testing by setting up a loop to constantly publish a message. When running just one thread 52000 messages where added to the queue in 60 seconds. When 5 threads where run, each doing the same loop, only 8000 messages where added to the queue in 60 seconds.

Why when running 5 threads is the performance much worse? Should each thread have its own bus instance?

Answer 1

A single bus instance will create a single connection to RabbitMQ. And that is typically enough for most situations. If you want to scale out the connections, currently additional bus instances would need to be created.

However, you should be able to publish at a fairly high rate, particularly if you have multiple threads which play nicely with the TPL (Task Parallel Library). You can also combine multiple Publish calls into a single Task.WhenAll(), to avoid the latency between each message.

A good test is to use the MassTransit-Benchmark project to get an idea of the throughput of your broker. Typically I see anywhere from 8000-12000 messages per second published and consumed through RabbitMQ (non-durable, with publisher confirmation disabled). Adding durability, and confirmation, can slow down single-threaded publishing but shouldn't impact multiple source threads significantly. The durable writes to disk can slow things down for sure.

I'd be curious of the code you used where multiple threads were slower. I find it best to use Task.Run() rather than Thread.Create() (or whatever the syntax is now, I never use threads anymore).