Strange intermittent SignalR failures when using Azure Servicebus Backplane

Question

We are seeing strange behavior in our development and production environments when using SignalR 2.2.0 and using the Microsoft ASP.NET SignalR Service Bus Messaging Backplane with certain Azure Service Bus Instances. Some service buses seem to corrupt and become clogged, and we see the problems described below.

First, here is my OWIN startup code:

public void Configuration(IAppBuilder app)
{
    string connectionString = System.Configuration.ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
    GlobalHost.DependencyResolver.UseServiceBus(connectionString, "MyApplicationName");

    // Branch the pipeline here for requests that start with "/signalr"
    app.Map("/signalr", map =>
    {
        map.UseCors(CorsOptions.AllowAll);
        var hubConfiguration = new HubConfiguration
        {
            //EnableJSONP = true,
            EnableDetailedErrors = true
        };
        map.RunSignalR(hubConfiguration);
    });
}

The symptoms of our problem is an intermittent failure for SignalR to connect using any transport. Performance is slow compared to running with no backplane, and with verbose logging enabled on the SignalR client, I see the message "SignalR: webSockets transport timed out when trying to connect." SignalR then tries to go through the rest of the transports (Forever frames, long polling) and then gives up.

Here's the kicker: for some of our service bus instances, the performance is rock solid and we never have issues. Other service bus instances cause the problems described above.

Why do we have multiple service buses? We only use one for our app, but developers each have a service bus instance to play with. It keeps me up at night that an Azure Service Bus has corrupted and I don't know why.

Questions:

Has anyone else experienced this problem?
Have you seen Service Bus instances corrupt or misbehave with SignalR or other applications?
What could account for this behavior?

I am also currently experiencing this exact same problem. Certain service bus instances fail to connect from SignalR while others function without any issue at all. Did you ever arrive at a solution to this problem? — Pepto

Rob Reagan Rob Reagan · Accepted Answer · 2017-04-28T14:45:52

This is an old post, but we had a very similar problem that had us pulling our hair out. This occurred when we were using SignalR with a Service Bus backplane.

In our exception logs, we found the following:

The X.509 certificate CN=servicebus.windows.net is not in the trusted people store. The X.509 certificate CN=servicebus.windows.net chain building failed. The certificate that was used has a trust chain that cannot be verified. Replace the certificate or change the certificateValidationMode. A certificate chain could not be built to a trusted root authority.

The fix was to add the following line of code in our app startup. For us, that was global.asax.cs:

ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Http;

After that, our service bus backplane and SignalR ran perfectly.

For a much more in-depth discussion of what's going on, check out this SO post.

Strange intermittent SignalR failures when using Azure Servicebus Backplane

1 Answers