Cannot get NServiceBus to not throw exceptions (MessageQueueException, SqlException, TimeoutPersisterReceiver)

Question

I have a very frustrating NServiceBus problem that I cannot seem to figure out. I am hoping that you guys can shed some light on the situation.

I am currently using NServiceBus.Core v5.0 and NServiceBus.Host v6.0 and running it in Unobtrusive Mode.

It seems that no matter what configuration I use, I always get some kind of error. I will start with the configuration that produces the least problems:

Case 1 - Using custom assembly scanning:

public void Customize(BusConfiguration configuration)
    {
        var endpointName = typeof(EndpointConfig).Namespace;
        configuration.SetUniqueHostId(endpointName);

        configuration.UseSerialization<JsonSerializer>();
        configuration.UsePersistence<NHibernatePersistence, StorageType.Outbox>();

        configuration.AssembliesToScan(new List<Assembly>
        {
            GetType().Assembly,
            typeof(ICustomCommand).Assembly
        });

        configuration.Conventions()
            .DefiningCommandsAs(type => typeof(ICustomCommand).IsAssignableFrom(type));

        configuration.EnableDurableMessages();
        configuration.EnableInstallers();

        var container = ContainerInitializer.Container;
        configuration.UseContainer<AutofacBuilder>(c => c.ExistingLifetimeScope(container));
    }

The issues I have noticed here are the following:

1. When starting the NServiceBus host application and the persistence SQL database does not yet exist, no exception is thrown saying that the database cannot be found (it does in case 2).

2. I keep getting the following exception:

NServiceBus.Timeout.Hosting.Windows.TimeoutPersisterReceiver Failed to fetch timeouts from the timeout storage

Which ultimately results in the application crashing because when this error occurs too many times, ServiceBus decides that enough is enough and just throws a fatal exception.

3. Besides the issues above, the application runs perfectly receiving and processing messages... until the fatal exception occurs

Now, this one is a bit more difficult:

Case 2 - Using default assembly scanning:

public void Customize(BusConfiguration configuration)
    {
        var endpointName = typeof(EndpointConfig).Namespace;
        configuration.SetUniqueHostId(endpointName);

        configuration.UseSerialization<JsonSerializer>();
        configuration.UsePersistence<NHibernatePersistence, StorageType.Outbox>();

        // !! DISABLED !!
        // configuration.AssembliesToScan(new List<Assembly>
        // {
        //    GetType().Assembly,
        //    typeof(ICustomCommand).Assembly
        // });

        configuration.Conventions()
            .DefiningCommandsAs(type => typeof(ICustomCommand).IsAssignableFrom(type));

        configuration.EnableDurableMessages();
        configuration.EnableInstallers();

        var container = ContainerInitializer.Container;
        configuration.UseContainer<AutofacBuilder>(c => c.ExistingLifetimeScope(container));
    }

In this case the following issues occur:

When the persistence SQL database does not yet exist:

1. When starting the NServiceBus host application and the SQL database does not exist, an exception is throw - Expected behavior (This is positive)

After creating the persistence SQL database:

1. ServiceControl.Plugin.Nsb5.Heartbeat.Heartbeats|Unable to send heartbeat to ServiceControl: NServiceBus.Unicast.Queuing.QueueNotFoundException: Failed to send message to address: [Particular.ServiceControl@MYPCNAME]

2. Exception thrown: 'System.Messaging.MessageQueueException' in System.Messaging.dll Additional information: External component has thrown an exception.

3. 2017-09-15 16:25:45.6743|Warn|NServiceBus.Unicast.Messages.MessageMetadataRegistry|Message header 'SharedTemp.Interfaces.ICustomCommand' was mapped to type 'SharedTemp.Interfaces.ICustomCommand' but that type was not found in the message registry [...]

4. Exception thrown: 'System.Exception' in NServiceBus.Core.dll Additional information: Could not find metadata for 'Newtonsoft.Json.Linq.JObject'.

Now, exceptions 3 and 4 are particularly (no pun intended) odd since the NServiceBus documentation states:

By default all assemblies in the endpoint bin directory are scanned to find types implementing its interfaces so that it can configure them automatically.

And the "Newtonsoft.Json" and my "SharedTemp dll's" are indeed in the BIN folder, but NServiceBus does not seem to find them. As for point 1: NServiceBus does not create that queue for me, but it creates all the other queues that I need.

Finally the always requested app.config file:

    <?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="MasterNodeConfig" type="NServiceBus.Config.MasterNodeConfig, NServiceBus.Core"/>
    <section name="MessageForwardingInCaseOfFaultConfig" type="NServiceBus.Config.MessageForwardingInCaseOfFaultConfig, NServiceBus.Core"/>
    <section name="UnicastBusConfig" type="NServiceBus.Config.UnicastBusConfig, NServiceBus.Core"/>
    <section name="AuditConfig" type="NServiceBus.Config.AuditConfig, NServiceBus.Core"/>
  </configSections>
  <appSettings>
    <add key="NServiceBus/Persistence/NHibernate/dialect" value="NHibernate.Dialect.MsSql2012Dialect"/>
    <add key="NServiceBus/Outbox" value="true"/>
  </appSettings>
  <MessageForwardingInCaseOfFaultConfig ErrorQueue="error"/>
  <UnicastBusConfig>
    <MessageEndpointMappings />
  </UnicastBusConfig>
  <AuditConfig QueueName="audit"/>
  <connectionStrings>
    <add name="NServiceBus/Persistence" connectionString="Server=(LocalDB)\v11.0;Initial Catalog=NServiceBus;Integrated Security=true"/>
  </connectionStrings>
  <system.web>
    <membership defaultProvider="ClientAuthenticationMembershipProvider">
      <providers>
        <add name="ClientAuthenticationMembershipProvider" type="System.Web.ClientServices.Providers.ClientFormsAuthenticationMembershipProvider, System.Web.Extensions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" serviceUri=""/>
      </providers>
    </membership>
    <roleManager defaultProvider="ClientRoleProvider" enabled="true">
      <providers>
        <add name="ClientRoleProvider" type="System.Web.ClientServices.Providers.ClientRoleProvider, System.Web.Extensions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" serviceUri="" cacheTimeout="86400"/>
      </providers>
    </roleManager>
  </system.web>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.2"/>
  </startup>
  <MasterNodeConfig Node="localhost"/>
</configuration>

Does anyone have an idea about any of this?

Answer 1

After a lot of searching I finally found the problem!

First of all I was using a in-house NuGet package that was supposed to help me with configuring NServiceBus. The package worked fine for other projects, but for mine not so well, since I was using JsonSerialization instead of the default XML serialization.

The first problem with the package is that it used the "INeedInitialization" interface to configure NServiceBus. In my code I would then call "IConfigureThisEndpoint" to enable the JsonSerialization. The issue here was, that when starting the NServiceBus host, it would fail to find the NewtonSoft.Json library. If I then added custom assembly scanning to my own configuration code, it would not trigger "INeedInitialization", causing an incomplete/incorrect configuration.

I assume it could not load the NewtonSoft.Json library because scanning was triggered in the code/namespace of the package? Maybe @Sean Farmer can answer this?

The second problem with the package is that it would add connection strings to the app.config, one for "NServiceBus/Persistence" and one for "NServiceBus/Persistence/NHibernate/Saga". I am not using Saga, so the connection string for that was not needed. Initially this was not a problem since I caught it the first time, but I completely forgot about it after reinstalling the package. Removing this again also seemed to make NServiceBus happier.

So, what ended up working? I removed the package and did the configuration myself with the following result:

public void Customize(BusConfiguration configuration)
    {
        var endpointName = typeof(EndpointConfig).Namespace;
        configuration.UniquelyIdentifyRunningInstance().UsingCustomIdentifier(endpointName);

        configuration.EnableOutbox();
        configuration.UsePersistence<NHibernatePersistence>();
        configuration.Transactions().DefaultTimeout(TimeSpan.FromMinutes(5.0));

        configuration.UseSerialization<JsonSerializer>();

        configuration.Conventions()
            .DefiningCommandsAs(type => typeof(ICustomCommand).IsAssignableFrom(type));

        configuration.EnableDurableMessages();
        configuration.EnableInstallers();

        var container = ContainerInitializer.Container;
        configuration.UseContainer<AutofacBuilder>(c => c.ExistingLifetimeScope(container));
    }

@Sean: Thank you for allowing me to contact you. Luckily it was not necessary

@Mike: Thank you for the input