Spring Cloud Data Flow with Azure Event Hub limitations?

1
votes

We plan to use Spring Cloud Data Flow on Azure Cloud using Azure EventHub as a messaging binder. On Azure EventHub, there are hard limits :

  • 100 Namespaces
  • 10 topics per namespaces.

The Spring Cloud Azure Event Hub Stream Binder seems to be able to configure only one namespace, so how can we manage multiple namespaces?

Maybe we should use multiple binders, to have multiple instances of the Spring Cloud Azure Event Hub Stream Binder?

Does anyone have any ideas? or documentation we did not find?

Regards

Rémi

1
spring-cloudazure-eventhubspring-cloud-dataflow

1 Answers

0
votes

Spring Cloud Data Flow and Spring Cloud Skipper support the concept of "platform accounts". Using that, you can set up multiple accounts, for each namespace or any other K8s clusters even. This opens a lot of flexibility to work around these hard limits in Azure stack.

We have a recipe on multi-platform deployments.

When deploying the streams from SCDF, you'd pick and choose the platform account (aka namespace or other configs), so automatically the deployed stream apps (with Azure binder in the classpath) would be running in different namespaces. Effectively, dodging the limits enforced in Azure.

The provenance tracking of where the apps run and the audit trail is automatically also captured in SCDF, so at any given time, you'd know who did what and in which namespace.