Azure web jobs - parallel message processing from queues not working properly

Question

I need to provision SharePoint Online team rooms using azure queues and web jobs. I have created a console application and published as continuous web job with the following settings:

            config.Queues.BatchSize = 1;

            config.Queues.MaxDequeueCount = 4;

            config.Queues.MaxPollingInterval = TimeSpan.FromSeconds(15);

            JobHost host = new JobHost();

            host.RunAndBlock();

The trigger function looks like this:

public static void TriggerFunction([QueueTrigger("messagequeue")]CloudQueueMessage message)
    {
        ProcessQueueMsg(message.AsString);
    }

Inside ProcessQueueMsg function i'm deserialising the received json message in a class and run the following operations:

I'm creating a sub site in an existing site collection;
Using Pnp provisioning engine i'm provisioning content in the sub site (lists,upload files,permissions,quick lunch etc.).

If in the queue I have only one message to process, everything works correct.

However, when I send two messages in the queue with a few seconds delay,while the first message is processed, the next one is overwriting the class properties and the first message is finished.

Tried to run each message in a separate thread but the trigger functions are marked as succeeded before the processing of the message inside my function.This way I have no control for potential exceptions / message dequeue.

Tried also to limit the number of threads to 1 and use semaphore, but had the same behavior:

private const int NrOfThreads = 1;
private static readonly SemaphoreSlim semaphore_ = new SemaphoreSlim(NrOfThreads, NrOfThreads);

//Inside TriggerFunction

           try
            {
                semaphore_.Wait();

                new Thread(ThreadProc).Start();
            }
            catch (Exception e)
            {
                Console.Error.WriteLine(e);
            }

public static void ThreadProc()
        {
            try
            {
                DoWork();
            }
            catch (Exception e)
            {
                Console.Error.WriteLine(">>> Error: {0}", e);
            }
            finally
            {
                // release a slot for another thread
                semaphore_.Release();
            }
        }

        public static void DoWork()
        {
            Console.WriteLine("This is a web job invocation: Process Id: {0}, Thread Id: {1}.", System.Diagnostics.Process.GetCurrentProcess().Id, Thread.CurrentThread.ManagedThreadId);
            ProcessQueueMsg();
            Console.WriteLine(">> Thread Done. Processing next message.");
        }

Is there a way I can run my processing function for parallel messages in order to provision my sites without interfering?

Please let me know if you need more details.

Thank you in advance!

mathewc mathewc · Accepted Answer · 2017-03-16T17:47:00

You're not passing in the config object to your JobHost on construction - that's why your config settings aren't having an effect. Change your code to:

JobHost host = new JobHost(config);
host.RunAndBlock();

Azure web jobs - parallel message processing from queues not working properly

1 Answers