"Unable to find assembly" after upgrading NServiceBus

Question

For our upcoming release, we have upgraded from NServiceBus 3.0.3 to 3.2.7. We have not had any problems during local or QA tests, but when testing in the staging environment, we cannot start the IIS-hosted web services which reference NServiceBus.

The staging machines use the same software as QA; Windows 2008R2, IIS7.5

The eventlog has two errors. First is event 1026 (.NET runtime)

Application: w3wp.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: NServiceBus.Unicast.Queuing.FailedToSendMessageException Stack: at NServiceBus.Unicast.Queuing.Msmq.MsmqMessageSender.NServiceBus.Unicast.Queuing.ISendMessages.Send(NServiceBus.Unicast.Transport.TransportMessage, NServiceBus.Address) at NServiceBus.Unicast.UnicastBus.SendSubscribeMessageWithRetries(NServiceBus.Address, NServiceBus.Unicast.Transport.TransportMessage, System.String, Int32) at NServiceBus.Unicast.UnicastBus+<>c_DisplayClass8.b_7(System.Object) at System.Threading.ExecutionContext.runTryCode(System.Object) at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

Followed by event 1326 (ASP.NET 4.0.30319.0)

An unhandled exception occurred and the process was terminated.

Application ID: DefaultDomain

Process ID: 10180

Exception: System.Runtime.Serialization.SerializationException

Message: Unable to find assembly 'NServiceBus.Core, Version=3.2.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c'.

StackTrace: at System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly() at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo assemblyInfo, String name) at System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA, Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo, SizedArray assemIdToAssemblyTable) at System.Runtime.Serialization.Formatters.Binary._BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record) at System.Runtime.Serialization.Formatters.Binary._BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) at System.AppDomain.Deserialize(Byte[] blob) at System.AppDomain.UnmarshalObject(Byte[] blob)

I have checked the bin folders, and they do contain the correct version of NServiceBus.dll and NServiceBus.Core.dll (and both are the same version, 3.2.7).

I'm feeling pretty stumped, think I've checked the obvious things. Could it be that it is trying to load several versions of NServiceBus (ie some reference that is not updated?). However, that should have been a problem already in QA?

The exception says "3.2.0.0" and you say "3.2.7". Did you try to set up assembly rebinding so that any reference to 3.2.0.0 is resolved with 3.2.7? — Wiktor Zychla
@WiktorZychla: I believe Nservicebus uses a versioning scheme such that 3.2.* must be compatible with each other, and therefore tag their binaries with just 0 for the remaining positions, semver.org. The binaries we got from NuGet are labeled as 3.2.7 in nuget, but examining them in windows says version 3.2.0.0 — carlpett
Ok. Yet another clue would be to catch appdomain's AssemblyResolve. If their new version relies on yet another assembly which is not found then the serializer could have problems but you don't see the real cause. Assemblyresolve would then fire to try to load the missing assembly. Also check the inner exception. — Wiktor Zychla
Did you find an answer to this? Just updated from 3.2.8 to 3.3.3 via Nuget and now IIS complains with {"Could not load file or assembly 'NServiceBus, Version=3.2.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)":"NServiceBus, Version=3.2.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c"} — mellodev
@mellodev: As I recall, it was due to someone adding a reference to NServiceBus.Host in an WCF application, have you checked that? Been some time though, so I may be mixing up the silly errors... If it's not that, I could have a look when I get back to work in two weeks. — carlpett

Iain Iain · Accepted Answer · 2014-08-24T09:33:04

While an old question, I hit this issue and I have a workaround.

First enable fusion logging. Make sure the process can write to the log folder.

Next, check where process is looking. In my case it was only searching file:///c:/windows/system32/inetsrv/ so I added the dll NServiceBus.Core.DLL there, and now I can see the real issue.

My guess is an unhandled exception occurred on a background thread (the first error), that was then serialised and passed to the main thread, which tried to deserialise it, but it could not find the assembly (the second error). This second error killed the app pool and meant I lost the real error.

This workaround only address the second error, not the original issue. Having said that, so far this has been enough for me to get the app working again and find the real error which for me was MSMQ was out of memory. So I restarted MSMQ and it has started working. I then removed NServiceBus.Core.DLL from the new location and the app is still working.

Logging output:

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll

=== Pre-bind state information ===
LOG: User = IIS APPPOOL\TenBagsFull
LOG: DisplayName = NServiceBus.Core
 (Partial)
WRN: Partial binding information was supplied for an assembly:
WRN: Assembly Name: NServiceBus.Core | Domain ID: 1
WRN: A partial bind occurs when only part of the assembly display name is provided.
WRN: This might result in the binder loading an incorrect assembly.
WRN: It is recommended to provide a fully specified textual identity for the assembly,
WRN: that consists of the simple name, version, culture, and public key token.
WRN: See whitepaper http://go.microsoft.com/fwlink/?LinkId=109270 for more information and common solutions to this issue.
LOG: Appbase = file:///c:/windows/system32/inetsrv/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = w3wp.exe
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: No application configuration file found.
LOG: Using host configuration file: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet.config
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///c:/windows/system32/inetsrv/NServiceBus.Core.DLL.
LOG: Attempting download of new URL file:///c:/windows/system32/inetsrv/NServiceBus.Core/NServiceBus.Core.DLL.
LOG: Attempting download of new URL file:///c:/windows/system32/inetsrv/NServiceBus.Core.EXE.
LOG: Attempting download of new URL file:///c:/windows/system32/inetsrv/NServiceBus.Core/NServiceBus.Core.EXE.
LOG: All probing URLs attempted and failed.
Running under executable  c:\windows\system32\inetsrv\w3wp.exe
--- A detailed error log follows.

"Unable to find assembly" after upgrading NServiceBus

2 Answers