Event Hub input binding for Azure Functions

1
votes

I have an Azure Function with an input binding to an Event Hub.

public static async Task Run(TraceWriter log, string eventHubMessage)

When the function is triggered, how many messages does it receive per execution by default?

Is it 1 execution = 1 message?

I have read the documentation and understand you can set these properties in the function's host.json file:

"eventHub": {
  // The maximum event count received per receive loop. The default is 64.
  "maxBatchSize": 64,
  // The default PrefetchCount that will be used by the underlying EventProcessorHost.
  "prefetchCount": 256
}

Does maxBatchSize mean I will receive 64 messages in 1 execution?

2
c#azureazure-functionsazure-eventhub

2 Answers

2
votes

@Mikhail is correct. I'd just like to add the following:

  1. If you use the default EventHub-Trigger C# template, the Function created will process 1 message per execution.

  1. If you need each execution to process in batches, change the following:

    a. In function.json, add the property "cardinality":"many" as shown here.

    b. In run.csx, modify Function signature and process messages in a loop, e.g.,

    public static async Task Run(TraceWriter log, string[] eventHubMessages) { foreach(string message in eventHubMessages) { // process messages } }

  2. The host.json configuration you specified in the question allows you to experiment with the correct batch size and prefetch buffer to meet the needs of your workflow.

Additional comments:

  1. Under the Consumption Plan, a Function is currently allowed a max default 5-minute execution time (configurable up to 10 mins --Added on 11/30/2017). You should experiment with the maxBatchSize and prefetchCount setting to ensure that a typical execution of the batch will complete within the timeframe.

  1. The prefetchCount should be 3-4 times the maxBatchSize.

  1. Each Function host instance is backed by a single EventProcessorHost (EPH). EPH uses a checkpointing mechanism to mark the last successfully processed message. A Function execution could terminate prematurely due to uncaught exceptions in the Function code host crashing, timeout or partition lease lost, resulting in an unsuccessful checkpoint. When the Function execution restarts again, the batch retrieved will have messages from the last known checkpoint. Setting a very high value for maxBatchSize will also mean that you must re-process a large batch. EventHub guarantees at-least-once delivery but not at-most-once delivery. Azure Functions will not attempt to change that behavior. If having only unique messages is a priority, you will need to handle de-duplication in your downstream workflows.
3
votes

By default it's going to be 1 by 1 processing, but you can do batches too. Change the signature of your function to 

public static async Task Run(TraceWriter log, string[] eventHubMessages)

(if you change the name like I did, rename the binding parameter too)

Reference github issue.