Azure Web Jobs and SignalR memory leak

Question

So I have some webjobs that occasionally connect to a signalr hub and broadcast a message. Below is just an example of one, in this case its a simple web job for development with the TimerTrigger attribute that is set to run continuously every 20 seconds. Like that shown in the code below.

    public static void Main()
    {

        JobHostConfiguration config = new JobHostConfiguration();
        config.Tracing.ConsoleLevel = TraceLevel.Verbose;
        config.UseTimers();
        if (config.IsDevelopment)
        {
            config.UseDevelopmentSettings();
        }
        var host = new JobHost(config);
        host.RunAndBlock();
    }

    public static void ProcessPush([TimerTrigger("00:00:20", RunOnStartup = true)] TimerInfo timerInfo, TextWriter log)
    {
        // Send a signalr message to the Hub
        try
        {            
            SendMessageToHub(log);
        }
        catch (Exception e)
        {
            log.WriteLine($"WebJob Push Exception: {e.Message}");
        }
    }

    private static async Task SendMessageToHub(TextWriter log)
    {
            var hub = new HubConnection(CloudConfigurationManager.GetSetting("MyWebSite"));
            var proxy = _hub.CreateHubProxy("MyHub");

            log.WriteLine("WebJob Push: Sending message to SignalR Hub.");
            if (_hub.State == Microsoft.AspNet.SignalR.Client.ConnectionState.Disconnected)
            {
                await _hub.Start();
            }
            await _proxy.Invoke("BroadcastMessage");
            log.WriteLine("WebJob Push: Sent message to SignalR Hub.");
    }

There is always an increment in memory on the server hosting the website and signalr hubs. When investigating the IIS logs on the website, it seems to have surge / batch of POST messages come to the website at the same second, using long polling. It then waits a little while and then gets bombarded with another batch of messages. By the way, this drives the CPU on the IIS Server crazy as well. At the bottom of this post is an example of the IIS log entries.

I'd like to be able to send a signalr message from the web job in a consistent manner with a regular pulse message (we want to expand on the web job - please excuse the fact it runs on a timer for now).

Kind regards,

Stefan

Example IIS Log entries - notice they have all come in within the same second, instead of 20 seconds apart:

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=TBbNVDpndk0riu8UvVzbGJrWjYIo7eMLcP4lk7ABV74OBMbZRTJrCRL1bzsPxpd1Tyle2rS3tV2JJrigninhu880ml51Xers76PPDX0Hf97dTBYR4k%2BVc2V9KAmiGt0p&messageId=d-80E0087-B%2C7D%7CEz%2C0%7CE0%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 5343

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=KYwQXpNrPIU21NXMa0So5u42EwXTcMlGyLqL3tetx4WfOtTunHLclG%2BhPd%2BcPeZPmfe6KKvQL13XIU1W5fApuTv0XN5XFPoNUmyBjhhISoqodwcZeu3QKmkbaXcpHMtE&messageId=d-80E0087-B%2C7D%7Cav%2C0%7Caw%2C2 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 10641

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=nO%2BPZ8M5JJOpiobpJUV5%2FZvQyEKYjp%2FOuqQ%2F0Bkq05TKRJZfeI%2FD%2BxRyPC7EsAAjXVqJr05PksorlMWrXocGkskfVsLU2Qvtx%2Fi1O8hU5lNz4KcoSc%2Bkv%2BlDpr2AZBLv&messageId=d-80E0087-B%2C7D%7CFB%2C0%7CFC%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 18282

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=wiGSRiNHdd7crhkcAMd%2FWy%2F3qGRZ5WdBm%2BdbR3b7aTbtpB8aaBGDil%2FqAWha6Si5eEohsUmCxAU4Pkefy%2BNoxoG9fgYC4R66ErXIShyBUcsNLWo1AyH5zGDk7bFvme3E&messageId=d-80E0087-B%2C7D%7CE9%2C0%7CE_%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 11360

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=hEJ1b0%2Bz2eeyC8IvYmOV3ffZ%2FAFQiQpEnJLUmCZTEVDLwcgOqhyQbQnu0R29sazp6BxcK4WsDhSbEdg2Sh4wMBSZjQtKMzASr2Fa2eY2HGgoVJcfDOMixQX2FCqfa%2BmP&messageId=d-80E0087-B%2C7D%7CFD%2C0%7CFE%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 15798

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=2UsU63IHgaNO%2BBYmoamsKxFq7Vv3uaGigvR1NrGnntVnAbTg2C0%2BVXZnA9aT8siqpkBv%2Fo8avvvNTSBfQD77IspaO6jOnSU8rXMXDU2Vr6ojkWr%2Fwt1LFsdNy3%2BHpDGC&messageId=d-80E0087-B%2C7D%7CEv%2C0%7CEw%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 11844

etc, etc

UPDATE - Explicitly stopping the hub connection seems to have dealt with the orphan clients (or superfluous clients for the same web job client). In particular, adding _hub.Stop(); after invoking the proxy.

The poll requests are from different clients (each has a different connectionToken). It only means that the server sent messages to clients on the existing poll requests and clients have to re-establish new polls to get new messages if any. If you want to avoid these requests use a more efficient transport - ideally websockets. (It's interesting that the client is actually running long polling. I could imagine it is not using websockets since they are disabled by default and you might have forgotten to enable them but I don't know why it is not using serverSentEvents) — Pawel
Did you try to make your triggered function async and change the async void to async Task in the SendMessageToHub method signature ? — Thomas
Hi Thomas, sorry, I have edited the code example here. It was actually: private static async Task SendMessageToHub(TextWriter log) — Stefan Zvonar
Hi Pawel, the website definitely has web sockets enabled. Most of our other signalr traffic is done via ws. However, the web job client does seem to be doing long polling. — Stefan Zvonar
I'm beginning to think that we might need to do a .Stop() method on the hub connection after firing the message, to stop any long polling - as the client is not interested in receiving anything, just firing messages. The signalr server must not be getting a disconnect signal when the webjob completes ... (?) — Stefan Zvonar

Stefan Zvonar Stefan Zvonar · Accepted Answer · 2016-11-23T22:23:46

So as obvious as it may sound to some, the answer to this was to ensure that we stop the hub connection after sending the message to it. Below is the code with the additional line surrounded by comments.

It would seem that unlike a browser client, whereby the client gets disconnected after a while of inactivity. The webjob client sticks around, and continues to poll the website. So each time the webjob is triggered (via a timer or via reading a message off an azure queue or topic) then it keeps spawning a new client and retaining that connection.

In a webjob, you must explicitly stop the connection, otherwise the memory and CPU will gradually go nutbags.

private static async Task SendMessageToHub(TextWriter log)
{
        var hub = new HubConnection(CloudConfigurationManager.GetSetting("MyWebSite"));
        var proxy = _hub.CreateHubProxy("MyHub");

        log.WriteLine("WebJob Push: Sending message to SignalR Hub.");
        if (_hub.State == Microsoft.AspNet.SignalR.Client.ConnectionState.Disconnected)
        {
            await _hub.Start();
        }
        await _proxy.Invoke("BroadcastMessage");
        /////////////////////////////////////////////////////////////
        // Stopping the hub connection is necesssary in a web job  //
        _hub.Stop();   
        /////////////////////////////////////////////////////////////
        log.WriteLine("WebJob Push: Sent message to SignalR Hub.");
}

Azure Web Jobs and SignalR memory leak

1 Answers