Azure Web Jobs / Queue - DecoderFallbackException

Question

I am using the Azure Web Jobs Feature. Here is what I do.

Step 1

• I got an Web Api Project which receives always the same string from the same client application.
• I have a simple class called SimpleClass which has just one property with always the same string from the same client application. (The string is APA91bELkr6CyBmqLbWomwkI2zw_GkXGVsblYH60l4hERXw9ZkCcXufjJM_7IZXI5_Ry9aze6AhYRVzBfl6CYq0kxrdV4ViPkW5hK2Rd2HlsZCDfhnOc3PGLt_SzIMjfbMRug_eK_di2YbJTA6weczoTyb-dKuvnwg)
• I serialize it using JsonConvert into a JSON string
• I write this into the Azure Storage Queue

Step 2

• I have create a very simple Azure Web Job Demo Job to process the queue
• The Job deserialize the object

But I got an exception sometime (only sometimes, this sucks)

Unhandled Exception: System.Text.DecoderFallbackException: Unable to translate bytes [FF] at index 4 from specified code page to Unicode.
at System.Text.DecoderExceptionFallbackBuffer.Throw(Byte[] bytesUnknown, Int32 index)
at System.Text.DecoderExceptionFallbackBuffer.Fallback(Byte[] bytesUnknown, Int32 index)
at System.Text.DecoderFallbackBuffer.InternalFallback(Byte[] bytes, Byte* pBytes)
at System.Text.UTF8Encoding.GetCharCount(Byte* bytes, Int32 count, DecoderNLS baseDecoder)
at System.String.CreateStringFromEncoding(Byte* bytes, Int32 byteLength, Encoding encoding)
at System.Text.UTF8Encoding.GetString(Byte[] bytes, Int32 index, Int32 count)
at Microsoft.WindowsAzure.Storage.Queue.CloudQueueMessage.get_AsString()
at Microsoft.Azure.Jobs.QueueCausalityHelper.GetOwner(CloudQueueMessage msg)
at Microsoft.Azure.Jobs.Host.Runners.Worker.GetFunctionInvocation(FunctionDefinition func, RuntimeBindingProviderContext context, CloudQueueMessage msg)
at Microsoft.Azure.Jobs.Host.Runners.Worker.MyInvoker.Microsoft.Azure.Jobs.ITriggerInvoke.OnNewQueueItem(CloudQueueMessage msg, QueueTrigger trigger, RuntimeBindingProviderContext context)
at Microsoft.Azure.Jobs.Host.Triggers.PollQueueCommand.TryExecute()
at Microsoft.Azure.Jobs.LinearSpeedupTimerCommand.Execute()
at Microsoft.Azure.Jobs.IntervalSeparationTimer.RunTimer(Object state)
at System.Threading.TimerQueueTimer.CallCallbackInContext(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.TimerQueue.FireQueuedTimerCompletion(Object state)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

Here is my code

Web API Controller

public class TestController : ApiController
{
    public HttpResponseMessage Post(Model model)
    {
        // Retrieve storage account from connection string.
        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
            CloudConfigurationManager.GetSetting("StorageConnectionString"));

        // Create the queue client.
        CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

        // Retrieve a reference to a queue.
        CloudQueue queue = queueClient.GetQueueReference("webjobsqueue");

        // Create the queue if it doesn't already exist.
        queue.CreateIfNotExists();

        // Create a message and add it to the queue.
        CloudQueueMessage message = new CloudQueueMessage(JsonConvert.SerializeObject(new SimpleClass { SimpleStringProperty = Model.Value }));
        queue.AddMessage(message);

        return new HttpResponseMessage(HttpStatusCode.OK);
    }
}

Web Job

class Program
{
    static void Main(string[] args)
    {
        var host = new JobHost();
        host.RunAndBlock();
    }

    public static void WaitForMessageInQueue([QueueTrigger("webjobsqueue")]
                                       string message)
    {
        var simpleClass = JsonConvert.DeserializeObject<SimpleClass >(message);
        Console.Out(simpleClass.SimpleStringProperty );
    }
}

###There is a post in the MSDN Forum with the same problem but no solution WebJobs Feedback

Answer 1

Not every sequence of bytes is valid UTF-8 text. If you're receiving a valid random Unicode/UTF-8 string, everything should work correctly. But if you're receiving random bytes (not UTF-8 text), this behavior is expected. CloudQueueMessage, the Azure Storage SDK's API for dealing with queue messages, kindly validates that it has valid text before handing it back to you. Apparently some of the other classes do not do that kind of validation and instead silently ignore invalid text.

I'd suggest validating Model.Value in your Web API action to ensure it's valid Unicode text before adding the message to the queue. I'd suggest using the result from model.Value.Normalize(), which says it will throw if the string contains invalid Unicode data.

Answer 2

I am using the latest version of the Webjob SDK and every thing work fine for me using your code:

I've created a simple model:

public class SimpleClass
{
    public string SimpleStringProperty { get; set; }
}

A class Function:

public class Function
{
    public void WaitForMessageInQueue([QueueTrigger("webjobsqueue")] string message)
    {
        Console.Out.WriteLine(message);
    }
}

And here is my sample web job program:

class Program
{
    // Please set the following connection strings in app.config for this WebJob to run:
    // AzureWebJobsDashboard and AzureWebJobsStorage
    static void Main()
    {

        // Retrieve storage account from connection string.
        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
            CloudConfigurationManager.GetSetting("MyConnectionString"));

        // Create the queue client.
        CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

        // Retrieve a reference to a queue.
        CloudQueue queue = queueClient.GetQueueReference("webjobsqueue");

        // Create the queue if it doesn't already exist.
        queue.CreateIfNotExists();

        // Create a message and add it to the queue.
        CloudQueueMessage message = new CloudQueueMessage(JsonConvert.SerializeObject(new SimpleClass { SimpleStringProperty = "APA91bELkr6CyBmqLbWomwkI2zw_GkXGVsblYH60l4hERXw9ZkCcXufjJM_7IZXI5_Ry9aze6AhYRVzBfl6CYq0kxrdV4ViPkW5hK2Rd2HlsZCDfhnOc3PGLt_SzIMjfbMRug_eK_di2YbJTA6weczoTyb-dKuvnwg" }));
        queue.AddMessage(message);

        var host = new JobHost();
        host.RunAndBlock();
    }
}