Immediate and scheduled business transactions - MSMQ/NServiceBus/MS Sql Service Broker/Windows Task Scheduler?

Question

I have this scenario, guys-

A typical large-scale CMS application in which the following business transactions appear:

• Users publish content immediately.
• Users schedule content to be published.
• Automatic process runs ,say, in 10 minutes interval, collects scheduled and ready (simply a status of the BO)content and sends it to publishing.

This is oversimplified description of the actual system. By saying publishing I mean a complex process consisting of 4 main sub-systems: building invocation arguments, generating html content by running another web app(asp.net), committing the transactional parties, and signaling the users about the publishing results. So the system looses ability to run the whole process in a single Mega-transaction, i.e. it's possible but not practical from the scalability/performance points of view.

There are several options for the sub-systems to communicate each other, like making use of MSMQ, SQL Service Broker, NServiceBus, and a chain of plain WCF one-way invocations (which is currently implemented). So, guys, I'm looking for some possibly more reliable and scalable solution for this processing because looks like the system will be getting more busy with growing number of users and , therefore, more content to be generated. Especially, once, mobile versions' support is requested by the clients.

One of my considerations is to queue all the users immediate requests using MSMQ where a dedicated WCF service will dispatch them next to the content generation module (web app). And so will do the windows scheduled task. I can't see how these messaging platforms could be useful but not residing them before the html generation. This really confuses me.

Can't figure out what technology will fit best for this on my own.Any help, considerations, sharing your experience will help me, guys.

Answer 1

You could use the saga capabilities in NServiceBus to model the process of publishing, including the scheduling part, as it has the ability to perform actions reliably after a certain delay with practically no overhead.

If you're happy enough to queue users immediate requests, then you could easily use NServiceBus as your API instead of cobbling something together yourself with MSMQ, WCF, and Windows Scheduled Tasks.

Answer 2

If all the players are SQL Server instances (or backed by such) then Service Broker has a big advantage. As is integrated in the SQL Engine it means that all those times when you have to engage in a local distributed transaction between the database and the message store (ie. every time you enqueue or dequeue a message) become a ordinary simple transaction since the database is the message store. This also gives advantages when you consider backup/restore since you can have a coherent backup for both the data and the messages (the database backup contains a backup of all the messages), you only needs only one solution for high availability/disaster recoverability: the database solution (clustering, mirroring) is also the messaging HA/DR story since the database is the message store. And built in activation eliminates the need to configure (and more importantly, monitor and failover in HA/DR failover cases) the external activation mechanisms. Not to mention that built-in activation has no latency while it can also adapt to spikes, something external scheduled activation can have problems achieving (external tasks must pool and have to balance the frequency of pooling with desired latency and consider spikes)

Answer 3

You will definitely require some business process handling.

Your immediate requests would translate to command messages. Once a command has been processed you could publish event messages that the next step in your process could pick up to continue the process. So you may not even need scheduling.

I have done something similar:

e-mail -> content engine -> document conversion -> workflow engine -> indexing -> workflow engine

No scheduling was involved. To get something done you send a command and when the command processing completes the endpoint publishes and event and whichever endpoint is subscribed to that event will receive a copy and know how to continue. I had a simple table structure keeping track of the data for the process along with the appropriate status.

For some parts of your system you may require more immediate processing. When you come across this you can install your endpoint service more than once and have one act as a 'priority' endpoint. As an example. In our document converter endpoint we would have to convert each e-mail body. Since we received thousands per day they would queue up quite a bit. This was done in the background with no real care for when it occurred. The users of the indexing web application, on the other hand, would need to convert certain documents immediately from the web-site. Sending to the same convert endpoint resulted in ad-hoc conversions waiting behind thousands of other conversion requests at times. So the response was not on. The solution was to simply install another converter instance (same binaries) and configure separate queues for the endpoint and then have all conversion request messages from the web servers route to the ad-hoc endpoint. Problem solved :)

As a side note: even when you do go with a web-service interface or WCF it may be a good idea to place that behind a service bus endpoint since the endpoint buys you quite a bit (retries / poison queues / etc.)

Since you seem to be in an evaluation stage you may want to take a look at my FOSS service bus: http://shuttle.codeplex.com/

This is what we used on the described system. There is a separate scheduler component should it be necessary (though we didn't use it on the project).