I encountered a strange issue with the startup of web roles. The full repro is quite complex, but I managed to find the root cause so I will put the simplified steps.
My webrole project depends on AssmeblyA which in turn depends on AssemblyB, Version=1.0. The webrole also depends directly on AssemblyB, but Version=2.0. (this all comes from different NuGet packages over which I don't have any control).
In the end, AssemblyB, Version=2.0 is getting copied to the \Bin folder. The website itself is working correctly because in the Web.config there is a binding redirect for AssemblyB to Version=2.0 (which is automatically being put there by nuget client).
However, when webrole is deployed to Azure, it fails to start because AssemblyB, Version=1.0 cannot be loaded.
I suspect this is because web role is first loaded from the Approot directory where web.config has no effect. I also found that I can workaround this problem by generating an app.config for the web-role as well and duplicating all binding redirects there. While this works, it's not very convenient to maintain such setup.
Does anyone know if this is a known issue with Azure web roles?
Tried with Azure SDk 2.4 and 2.5.1, all azure nuget packages up to date, no custom startup code whatsoever.
Update: Here is the exception obtained through IntelliTrace:
Unable to load the role entry point due to the following exceptions: -- System.IO.FileLoadException:
Could not load file or assembly 'Microsoft.Owin, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040) File name: 'Microsoft.Owin, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
And here is the stack trace:
CommonLanguageRuntimeLibrary!System.Reflection.RuntimeModule.GetTypes()
CommonLanguageRuntimeLibrary!System.Reflection.Assembly.GetTypes()
Microsoft.WindowsAzure.ServiceRuntime.dll!Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetRoleEntryPoint(System.Reflection.Assembly entryPointAssembly = {System.Reflection.RuntimeAssembly})
Microsoft.WindowsAzure.ServiceRuntime.dll!Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.CreateRoleEntryPoint(Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleType roleTypeEnum = IISWeb)
Microsoft.WindowsAzure.ServiceRuntime.dll!Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleType roleTypeEnum = IISWeb)
Microsoft.WindowsAzure.ServiceRuntime.dll!Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRole(Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleType roleType = IISWeb)
Microsoft.WindowsAzure.ServiceRuntime.dll!Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.AnonymousMethod()
CommonLanguageRuntimeLibrary!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext = {unknown}, System.Threading.ContextCallback callback = {unknown}, object state = {unknown}, bool preserveSyncCtx = {unknown})
CommonLanguageRuntimeLibrary!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext = {unknown}, System.Threading.ContextCallback callback = {unknown}, object state = {unknown}, bool preserveSyncCtx = {unknown})
CommonLanguageRuntimeLibrary!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext = {unknown}, System.Threading.ContextCallback callback = {unknown}, object state = {unknown})
CommonLanguageRuntimeLibrary!System.Threading.ThreadHelper.ThreadStart()
The error occurs because my web role project depends on Owin Version=3.0.1.0, but another DLL (which I get through nuget) depends on Owin Version=3.0.0.0. Hence the manifest mismatch.
So it seems like service runtime is still loading my assemblies in WaIISHost.exe just to check whether I do have RoleEntryPoint or not (I don't). Just as before, as soon as I put myassembly.dll.config with binding redirects copied from web.config into E:\approot\bin, everything starts correctly.