MongoDB as a queue, capped collection + tailable cursor

6
votes

(asked on mongo-user discussion group https://groups.google.com/d/topic/mongodb-user/i1ge4bNiMgM/discussion)

Hi, i'd like to use MongoDB as a message queue and thought about using a capped collection + tailable cursor for that purpose. I know that the tailable cursor can be used for accessing documents in the insertion order but

  1. can I use any sort order for the cursor?
  2. does a tailing cursor work with a filter query (to skip some documents I don't want to process)? If so, how does it handle insertion of new documents - will I be notified only if the new document matches the query criteria?
  3. Are tailing cursors fully supported by C# driver?
  4. I'd like my queue to guarantee single delivery - I mean if two clients try to read messages from the same collection they shouldn't be able to 'consume' the same message. I think this could be achieved with findAndModify, but how to do that with tailing cursor?

Thanks RG

1
c#mongodbmessage-queue

1 Answers

9
votes

There was a presentation from AOL/About.me on "MongoDB as message queue" at the Silicon Valley MUG in April, 2012 that you may find helpful in planning your approach.

  1. can I use any sort order for the cursor?
  2. does a tailing cursor work with a filter query (to skip some documents I don't want to process)? If so, how does it handle insertion of new documents - will I be notified only if the new document matches the query criteria?

A tailable cursor tails the end of a capped collection and only in natural order. If you want a sort order you need to use an indexed query instead.

Please have a read of the Create Tailable Cursor documentation page for more information.

Are tailing cursors fully supported by C# driver?

Yes, tailable cursor support was implemented in the 1.1 C# driver (see: code example).

I'd like my queue to guarantee single delivery - I mean if two clients try to read messages from the same collection they shouldn't be able to 'consume' the same message. I think this could be achieved with findAndModify, but how to do that with tailing cursor?

There are a few approaches:

  • Have a single reader with the tailable cursor distribute tasks to worker threads (i.e. based on worker queue depth or round-robin). This will be more efficient than having multiple readers, but is potentially a single point of failure.
  • (suggested by @Contango in comment below) Have multiple readers, but use optimistic concurrency to ensure that only one worker "owns" each task. This will result in more I/O, but may simplify your design if the workers are also tailing directly.

For more information on atomic operations, see Isolate Sequence of Operations in the MongoDB manual.