Azure WebJob Dies Silently - Nothing in Any Log

1
votes

I have an Azure WebJob with a QueueTrigger that had been working. Now it has a high percentage of failed messages. The failures are going to the poison message queue, but no other errors are reported. The failure does not appear in the webjobs dashboard, no errors appear in the output window. If I run the webjob locally in debug mode, same thing. All of the successful message show up and show as succeeded. The failures are simply ghosts. They seem to simply go straight to poison. All of the messages are based on the same POCO class and nothing in the message itself looks malformed. I have had issues with this webjob in the past, but it has always shown errors somewhere. I cannot find any message to point me in the right direction.

Update: I took the webjob and striped out almost all of the code...just to see what would happen. It worked for every message. I don't have any try/catch blocks in my code right now that would stop exceptions or other errors from showing themselves. Why does the job die silently (but only a portion of the time)?

1
azureazure-webjobsazure-webjobssdk
Have you set the Always On flag for the web app to true? Can you post the code for method which initializes the JobHost? – Sandesh
I have the same issue. I suspect it could be due to memory leaks across repeated runs. I did once get an out of memory exception but it has failed 10s of times without giving that error. I am on the consumption plan, so it shouldnt theoretically error for out of Memory anyway; it should just charge me for it – Dessus
Are you creating any X509Certificate2 objects? I had problems with Azure web apps silently failing and restarting the app pool because I wasn't passing the correct parameters to the constructor. – Jon Sagara
@Dessus: It certainly could be a compound memory leak issue. I will need to dig more. I am now revisiting this issue because V2 of my app is to be released soon. Jon: No cert objects. One thing that I thought could be the issue is blob storage. My jobs use an uploaded file as part of their processing. I upload the file to Azure storage and then queue a POCO message for the WebJob. Because the C# storage upload is async, it's possible that the file is not getting completely uploaded before the queue message processing starts and the WebJob can't get the full file. Will post updates. – Bryan Lewis

1 Answers

0
votes

I suspect it could have something to do with this:

Do I need to rethink how I write my functions? Overall, no, things work pretty much the same as before, with just one less setting to worry about. The same limits that applied before still apply: 1.5GB max memory and a 5-minute max execution time. The biggest impact that this change has is that you now are assured to be getting the right amount of resources and paying the least amount possible for your function executions.

That snippet was taken from this link.