NServiceBus for CQRS event publishing setup

Question

We're currently setting up a CQRS-based project. Each Web-project instance runs NEventStore, however all share the same EventStore persistence (underlying database).

Now we want to publish the stored events: not only to our ReadModel (one per Web-project instance) but also to additional event consumers (e.g. legacy applications, which also need to be informed in some way of events from our system, and many more).

Therefore we have x event consumers (= event handlers) and y event publishers but only one "real source" of events (the underlying EventStore database).

Q1: Is there now a best practice to connect those systems via Publish/Subscribe?

We thought about publishing the events from the EventStore via NServiceBus. All the consumers should subscribe for the event types they need - therefore each consumer also needs to subscribe at probably more than one publisher - Q2: Is this even possible? We've read that you cannot subscribe for the same event at multiple locations, see: David Boike "Once you subscribe to Message1 at QueueName@WebServer1, you will not also be able to subscribe to Message1 coming from QueueName@WebServer2."

Additional open questions: Q3: How to detect that a consumer has been shut down forever (if it has not successfully unsubscribed from the bus). -> Queue runs full?! How to differentiate this from the situation where it only lost the network connection for a bit?

Q4: What happens if the connection between subscription service and EventStore is not reliable and fails? Consumers register successfully at the subscription service but the EventStore is not aware of the new subscription and does not deliver messages...

Q5: In general: How does NServiceBus handle queues? What happens when the consumer is not reachable for a long amount of time (e.g. a few days)?

Answer 1

Q1: It kind of depends how you subscribe to messages. NServiceBus was build with the that there are business services (bounded contexts), which cannot share data. However in CQRS you have fat events that contain a lot of data, which are published so that subscribers can store the information in their read model. This read model is than used, for example, in the user interface. Either via a normal UI or via a composite UI (which gathers information from multiple business services). So it depends what you're looking for.

Q2: It is possible to subscribe to multiple events. However there is only one logical publisher of the event. So an event "CustomerWasBilled" cannot come from ComponentA and ComponentB. However you can subscribe to many different events and each publisher can of course have many different subscribers.

[Udi] A given publisher can be scaled out across multiple servers and NServiceBus will handle that for you - no need to subscribe to each machine.

Q3: I would think this is pure administration. If it should stay online and receive messages, queues will fill up. If it should be down/offline permanently and messages aren't processed, then you'd know pretty fast after the component was taken down. But I highly recommend documenting which subscribers and publishers are connected to each other and what happens if you change a message or a component.

[Udi] If it is an orderly shutdown, then it should include the unsubscribe. If it is a crash - ergo temporary, then you detect that with standard monitoring (WMI). In order to protect your queue from filling up, you can define when a message should be discarded using the TimeToBeReceived attribute.

Q4: This is handled in NServiceBus. It stores the subscriptions in a datastore, which can be SQL Server, RavenDB & InMemory. But it also keeps the subscriptions in memory and checks regularly if new subscriptions have been added. Should not be a problem with NServiceBus.

Q5: MSMQ is by itself reliable in nature. Of course if your entire server crashes and all messages were on the HD that fried, that's a problem. If a component is not reachable for a longer amount of days, it depends on SLA if this can happen. You can monitor message count in either incoming queue or error queue. Don't also forget DeadLetterQueue. But NServiceBus lets you also monitor how long it takes for messages to be processed.

When a component is not reachable for a long amount of time (e.g. days) than the business should decide what to do. If it's important messages get processed, introduce clustering, etc.

Hope this helps