Azure Worker Role runs on osFamily=“2” (Server 2008R2) but Fails on osFamily=“3” (Server 2012)

5
votes

We have an Azure worker role - .NET 4.0 - that runs fine when deployed to Azure on osFamily="2" (Server 2008R2). However, when we deploy with osFamily="3" (Server 2012) and no other code changes, the worker role continually recycles with the following 2 errors in the server event logs:

Faulting application name: WaWorkerHost.exe, version: 6.0.6002.18488, time stamp: 0x505cf7ca Faulting module name: KERNELBASE.dll, version: 6.2.9200.16384, time stamp: 0x5010ab2d Exception code: 0xe0434352 Fault offset: 0x00000000000189cc Faulting process id: 0xefc Faulting application start time: 0x01cdd4318f76d221 Faulting application path: E:\base\x64\WaWorkerHost.exe Faulting module path: D:\Windows\system32\KERNELBASE.dll Report Id: cf1810b0-4024-11e2-93ec-00155d4250e3 Faulting package full name: Faulting package-relative application ID:

Application: WaWorkerHost.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.InvalidOperationException Stack: at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.b__0() at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.ThreadHelper.ThreadStart()

Any ideas?

UPDATE ------------------ Found a 3rd exception:

Application: WaWorkerHost.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.Security.Cryptography.CryptographicException Stack: at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.b__0() at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.ThreadHelper.ThreadStart()

We are using a certificate in LocalComputer/My to decrypt connection strings. The certificate gets deployed successfully to OsFamily="2" (Server 2008R2) and has the Worker role user added to the Private Key (verified in RDP session). However, for OsFamily="3" (Server 2012) - the certificate doesn't get deployed, hence the crypto error. Following up with Azure support...UGH.

1
azure
Does control ever reach your OnStart()? – sharptooth
Did you try to debug the code in the emulator to find what method might be throwing an error? I am assuming you are using the Azure 1.8 SDK. – Bart Czernicki
Yes - app gets to OnStart and logs OnStart to Elmah (SQL). Yes - on the Azure 1.8 SDK and it runs fine in the emulator - no exceptions with either OSFamily selected. – viperguynaz
found the culprit - missed a 3rd exception: – viperguynaz
Well, wrap all stuff you have in OnStart() in one large try-catch, then catch and log the exception that is thrown first. – sharptooth

1 Answers

5
votes

From Azure Support Engineers:

After digging deeper, I found the difference: In OS Family 2, WaWorkerHost is running by a temporary account (with a GUID name) generated by Role initialization process, this account has permission to access certificate private key; In OS Family 3, WaWorkerHost is running by “NETWORK SERVICE” account, this account doesn’t have private key access permission.

I’m contacting production group, we will investigate this issue deeper. I will keep you posted.