Events dropped when integrating Azure Functions with Azure Event Hubs

Question

I am setting up an example where I have an Azure Function with a webhook that has an Azure Event Hub as a sink. There are two different consumer groups for this event hub, one of which is used by another Azure Function. The code doesn't do anything too fancy (excerpts are below), but I am observing an odd pattern. For every two messages published, only one appears to trigger the function once.

Publishing web hook:

module.exports = function (context, req) {
    var statusCode = 400;
    var responseBody = "Invalid request object";

    if (typeof req.body != 'undefined' && typeof req.body == 'object') {
        var eventInformation = req.body;

        context.log("Received event: " + eventInformation);

        context.bindings.outEvent = eventInformation;

        statusCode = 200;
        responseBody = "Event received.";
    }

    context.res = {
        status: statusCode,
        body: responseBody
    };

    context.done();
};

Receiving function:

module.exports = function (context, offerMadeEvent) {
    var connection = new Connection(dbConfig); // Tedious connection
    connection.on('connect', function(err) {
        context.log('Connection established.');
        // Somewhere in the database callbacks:
        context.done();
    });
};

Additionally, I have 2 partitions on my event hub. I am wondering if this is possibly related, but I do not see any partition-related configurations in docs.microsoft.com/en-us/azure/azure-functions/… so I'm not seeing a precise relationship. — Michael
Does having the Function App "always on" change things? When a Function App is on a consumption plan, then there are some known limitations on responsiveness (i.e.: delay in BlobTrigger getting triggered, etc). Perhaps we're seeing similar behavior here? — flyte
At the moment, I'm mostly just running the job with the CLI tools on my workstation. I have it logging on trigger (I understand that aggressive logging is not good in production) and that has been the basis. An example session (anonymized somewhat, of course) is available at pastebin.com/e3rfgiTq — Michael
@Michael, are you by any chance not using $Default the consumer group for 1 of the 2 Functions? There is a bug that is being actively fixed for non-default consumer groups. If you have not already done so, make at least one of the Functions use the $Default consumer group or just leave out the "consumerGroup" entry in your function.json file. Either approach will cause the system to use the default consumer group, which should trigger as expected. — Ling Toh

Ling Toh Ling Toh · Accepted Answer · 2017-03-29T23:47:02

Based on @Michael's the response from the "Comment" section above, the issue is related to a bug in the support for non-default consumer groups.

Background
A Function App instance is a process instance for all Functions inside the Function App. For a Function to execute, at least 1 Function App instance must be running.

Typically, a Function App instance is launched due to one of the following scenarios:

Function App is being accessed via Azure Portal - The UX workflow loads the Function App on behalf of the user, effectively forcing the Function App to come alive.
Function App instance is launched by internal service - There is an internal service that acts as the proxy listener for events on all triggers. It is responsible for listening for new events, launching a Function App instance if no-active instances exists, and scaling new instances when necessary.

For Functions that are created under the Consumption Plan, the Function App instance will stay alive for 5 minutes. After 5 minutes, the Function App instance will idle out. Once the "last" Function App instance idles out, if Scenario #1 does not occur, then the Function will only be triggered by Scenario #2.

Issue
For EventHub triggers, the internal service is currently only listening for events on the default consumer group. It does not detect when there are new events arriving for other consumer groups and will not wake up the Function App.

This is a known issue and is actively being fixed. You may track the issue at https://github.com/Azure/Azure-Functions/issues/230

Events dropped when integrating Azure Functions with Azure Event Hubs

1 Answers