Publish to Azure EventHub from AWS

Question

I am working on a use case wherein we collect data from multiple entities thru REST APIs and persist the on an RDBMS; this is an existing system. This stack is running on Windows Services on AWS (EC2 instances).

Now, we want to publish this data that is being persisted onto an Azure EventHub. A traditional way will be to use the Azure EventHub REST endpoint to publish the data from a Windows Service.

I do understand that Azure EventHub supports throughput of 1 MB/s Ingress. With the throughput that we are forecasting (~0.5 to 0.75 MB/s), we may not be using EventHub to its fullest capacity. Are there any other mechanisms that we can use for the above use case?

Answer 1

It sounds like the REST API on to Event Hub would be an adequate solution. Only when you go above the limits of the service does it become a problem from a scalability perspective. It is fine to have an Event Hub that is underutilised, even at 0.0001 MB/s. Note that you only get billed for the capacity you use, not for the service as a whole.

You do have some alternative options for ingesting the data, depending on what features the RDBMS has. Alternatively, your could push your incoming data on to a service bus (like Amazon SQS) and have it route to your Event Hub and to the RDBMS.

Crossing cloud providers in a high volume scenario may become expensive since you are paying for the ingress and egress out of the cloud data centres. If you could stick with one cloud provider you could save those costs.