nservicebus worker role in azure restarting due to databus configuration issue

Question

I have two worker roles setup as nservicebus endpoints (nservicebus 3.2.8) and both use AzureDataBus configurations for messages that are larger than the size allotted for azure queues. My employer does not allow placing credentials in the configuration so a IProvideConfiguration is used to configure the storage credentials.

This seems to work, however it appears that nservicebus is still trying to intialize the databus channel with a local development storage connection and this is causing the worker role to restart several times. Eventually it starts and picks up the correct configuration.

Am i doing something incorrect in regards to how i should be configuring the databus channel?

Here is the exception from the worker role's event log:

An unhandled exception occurred. Type: System.Exception Process ID: 2420
Process Name: WaWorkerHost
Thread ID: 6
AppDomain Unhandled Exception for role My.WorkerRole.Assembly_IN_1
Exception: Exception when starting endpoint, error has been logged. Reason: Unable to connect to the remote server
   at NServiceBus.Hosting.GenericHost.Start()
   at NServiceBus.Hosting.Azure.RoleEntryPoint.Run()
   at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.StartRoleInternal()
   at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.<StartRole>b__1()
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Inner Exception: Unable to connect to the remote server
   at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.get_Result()
   at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.ExecuteAndWait()
   at Microsoft.WindowsAzure.StorageClient.CloudBlobContainer.CreateIfNotExist(BlobRequestOptions options)
   at NServiceBus.DataBus.Azure.BlobStorage.BlobStorageDataBus.Start()
   at System.EventHandler.Invoke(Object sender, EventArgs e)
   at NServiceBus.Unicast.UnicastBus.NServiceBus.IStartableBus.Start(Action startupAction)
   at NServiceBus.Hosting.GenericHost.Start()

Inner Exception: No connection could be made because the target machine actively refused it 127.0.0.1:10000
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

Which promptly causes a process termination:

Application: WaWorkerHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Exception
Stack:
   at NServiceBus.Hosting.GenericHost.Start()
   at NServiceBus.Hosting.Azure.RoleEntryPoint.Run()
   at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.StartRoleInternal()
   at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.<StartRole>b__1()
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

Here is my EndPoint Configuration:

internal class EndpointConfig : IConfigureThisEndpoint, AsA_Worker, IWantCustomInitialization
    {
        #region IWantCustomInitialization Members

        public void Init()
        {
            Configure.With()
                     .DefaultBuilder()
                     .AzureConfigurationSource()
                     .MessageForwardingInCaseOfFault()
                     .AzureMessageQueue()
                     .JsonSerializer()
                     .AzureDataBus()
                     .AzureSubcriptionStorage()
                     .UnicastBus()
                     .DisableTimeoutManager()
                     .DisableSecondLevelRetries()
                     .IsTransactional(true)
                     .IsolationLevel(System.Transactions.IsolationLevel.ReadCommitted)
                     .PurgeOnStartup(false);
            }
}

Databus configuration override:

 public class ConfigOverride : IProvideConfiguration<AzureDataBusConfig>
    {

        AzureDataBusConfig IProvideConfiguration<AzureDataBusConfig>.GetConfiguration()
        {
            return new AzureDataBusConfig()
            {
                ConnectionString = "my storage key";
            };
        } 

    }

Yves Goeleven Yves Goeleven · Accepted Answer · 2013-02-04T20:30:04

Sounds like a timing problem in the initialization logic, basically you invoke AzureDataBus() before you config override is applied by nsb and you fall back to the default settings. Can you try putting the IWantCustomInitialization in a separate class (than it will run a bit later).

If this does not help, feel free to send me a small repro and I'll have a look at it!

PS: You can omit .With().DefaultBuilder() as that is the default anyway!

Kind regards, Yves

nservicebus worker role in azure restarting due to databus configuration issue

1 Answers