Azure Cloud Service Worker Role Crashing and Unhealthy

2
votes

I have a project with one Web role and one Worker role. Both my web and worker roles work as intended whenever I locally test it. (Note, although the local version uses the Azure Compute Emulator, the Service Bus queue and Message queue use the live cloud version.) However, the Cloud Service worker role has a tendency to crash during certain operations. It causes the instance to "Recycle" and the message gets stuck in the Azure Service Bus queue. It isn't until I locally run the cloud service that it successfully processes the message and then removes it from the queue. Even when the role works, Azure often tells me on the dashboard that the worker role is running "unhealthy."

At one point, the Azure portal printed this error:

Unhandled Exception: System.Reflection.ReflectionTypeLoadException

I've added exception handling in my project for this and tried again after deploying the update to the cloud, but the same thing keeps happening. When I added the exception handling to my project, I had to add the System.Reflection library to my worker role, but this confused me even more. I don't understand why I'm receiving an error for a call to a library that I did not even originally use in my project.

Anyone have any ideas on what this problem could be?

Update:

I did some manual debugging, and noticed that the exception is being called in the Run() of the Worker Role. The line of code causing the problem involves making a call to the database using Entity Framework. The specific line is:

Image imageEntity = db.Images.Find(ImgViewFromQueue.Id);

I'm wondering if the issue has to do with Entity Framework not being enabled or configured properly on the Azure server.

2
.netazureazure-storageazure-worker-roles
Can you check whether you are into any of the errors as listed here – Naveen Vijay
I just tried doing the things they mentioned on that page by setting all the references to true, and it didn't fix anything. – TheKarateKid
In your worker role are you using Linq or ADO.NET? – Ashwin Singh
Does control reach OnStart()? If so - wrap OnStart() and Run() implementations into try-catch and catch and log the exception with the call stack and all possible details. – sharptooth
@AshwinSingh I am using Entity Framework (I think with ADO.NET). – TheKarateKid

2 Answers

1
votes

I had similar issues running a worker role on OSFamily=3 (Server 2012). If you are running on 3 and can roll back to 2 you may find the worker role works just fine. I answered my question with help from Azure Support engineers - Azure Worker Role runs on osFamily="2" (Server 2008R2) but Fails on osFamily="3" (Server 2012)

The difference: In OS Family 2, WaWorkerHost is running under a temporary account (with a GUID name) generated by the Role initialization process, this account has special permissions; In OS Family 3, WaWorkerHost is running by “NETWORK SERVICE” account, this account is missing some permissions for worker roles.

If you have to have OSfamily 3 you can try running with elevated permissions. In your cscfg file, in the <WorkerRole> element, add the child:

<Runtime executionContext="elevated" />
0
votes

Not sure if you can enable the InelliTrace and re-deploy your package, then hopefully you will find the actual exception in your system. Just my 2 cents.