How to implement message queuing and handling in AWS with NServiceBus

4
votes

I am creating a new ASP MVC order application in the Amazon (AWS) cloud with the persistence layer at my local datacenter. I will be using the CQRS pattern. The goal of the project is high availability using Queue(s) to store and forward writes (commands/events) that can be picked up and handled asynchronously at my local datacenter. Then, ff the WAN or my local datacenter fails, my cloud MVC app can still take orders and just queue them up until processing can resume.

My first thought was to use AWS SQS for the queuing and create my own queue consumer/dispatcher/handler in my own c# application to process the incoming messages/events.

MVC (@ Amazon) --> Event/POCO --> SQS --> QueueReader (@ my datacenter) --> DB

Then I found NServiceBus. NSB seems to handle lots of details very nicely: message handling, retries, error handling, etc. I hate to reinvent the wheel, and NServiceBus seems like a full featured and mature product that would be perfect for me.

But on further research, it does NOT look like NServiceBus is really meant to be used over the WAN in physically separated environments (Cloud to my Datacenter). Google and SO don't really paint a good picture of using NServiceBus across the WAN like I need.

Can I do this?

MVC (@ Amazon) --> Event/POCO --> NServiceBus over WAN --> NServiceBus Handler(s) --> DB

How can I use NServiceBus across the WAN? Or is there a better solution to handle queuing and message handling between Amazon an my local datacenter?

2
c#nservicebuscqrsamazon-sqs
NServiceBus now has full support for SQS so you can do exactly what you originally planned of having code in your data center pick up messages sent by your MVC front-end in Amazon through it. Here's the documentation on SQS in NServiceBus: docs.particular.net/transports/sqs – Udi Dahan

2 Answers

6
votes

Using SQS as a transport for NServiceBus is an option, however you have to be aware of the trade offs as described here. This has been done with Azure queue storage, though I'm not aware of any great SQS implementations.

Another option is to create a VPN between your datacenter and an AWS VPC. This would allow direct MSMQ communication between AWS servers and your data center, provided you open the appropriate ports in the corresponding security group. There are some caveats with this approach. First, is regarding endpoint names. NServiceBus version 2.6 and below uses Environment.MachineName as the name of the endpoint, for which you would have to setup a proper DNS. I believe later versions use the machine's IP address. Perhaps a more important caveat is that a VPN makes your systems more coupled.

Yet another way, is to use the NServiceBus notion of a gateway. This however should be a logical business decision. A gateway is very similar to the regular transport but is usually has a different business context behind it.

4
votes

NServiceBus includes a Gateway component that handles bridging physically separated data centers.

http://docs.particular.net/nservicebus/gateway/

It basically moves the messaging to an HTTP channel and handles the retry logic and deduplication issues that you'd normally have with a web service.

If you download the full NServiceBus package (not just include it via NuGet) then you will see a folder full of samples and one of those covers usage of the Gateway, and that is a great way to get started.