How to log messages to Azure Event Hub from Azure API Management Service

Question

In one of the recent project, I need to add messages(>200kb) to Azure Event Hub through an endpoint exposed by Azure API Management Service. Then, the Stream Analytics job reads this message from Event Hub and writes it to the respective tables in SQL Server.

I was using "log-to-eventhub" policy to log the messages to event hub. But it has a size limitation associated with it, which is 200kb.

What would be the best approach to overcome this size limitation or should I consider a different way to log the payload to Event Hub? Any help is much appreciated.

Answer 1

Here is a limit about this described in official docs.

The maximum supported message size that can be sent to an event hub from this API Management policy is 200 kilobytes (KB). If a message that is sent to an event hub is larger than 200 KB, it will be automatically truncated, and the truncated message will be transferred to event hubs.

You could consider using Azure Event Hubs output binding for Azure Functions.

About How Function consume Event Hubs events, you could try using multiple parallel Functions instances under consumption plan.