Multiple Subscriptions in Azure Account Streaming to a single EventHub

Question

I want to stream the events from different subscription to a single eventhub on azure. At present I have configured eventhub to a single subscription and events are being streamed. I have a java client which consumes these events and stores it on my persistence layer. My java client looks like..

private void processUsingProcessorClient(){
        List<Disposable> subscriptions = null;
        try {

            EventHubConsumerAsyncClient eventHubConsumerAsyncClient = new EventHubClientBuilder()
                    .consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
                    .connectionString(CONNECTION_STRING, EVENT_HUB_NAME)
                    .credential("*******.servicebus.windows.net","maney-event-hub",createClientSecretCredential())
                    .buildAsyncConsumerClient();
            ReceiveOptions receiveOptions = new ReceiveOptions().setTrackLastEnqueuedEventProperties(true);
            List<String> block = eventHubConsumerAsyncClient.getPartitionIds().collectList().block();
            Iterator<String> iterator = block.stream().iterator();
            String partitionID = null;
            subscriptions = new ArrayList<>(block.size());
            while(iterator.hasNext()){
                partitionID = iterator.next();
                Disposable subscription = eventHubConsumerAsyncClient.receiveFromPartition(
                        partitionID,
                        EventPosition.fromOffset(0),receiveOptions).subscribe(PARTITION_PROCESSOR,ERROR_HANDLER);
                subscriptions.add(subscription);
            }
            System.in.read();
        }catch (Exception ex){
            ex.printStackTrace();
        } finally {
            if(subscriptions != null){
                subscriptions.forEach( subscrip -> {
                    subscrip.dispose();
                });
            }
        }
    }


    private final Consumer<PartitionEvent> PARTITION_PROCESSOR = partitionEvent -> {
        EventData event = partitionEvent.getData();
        PartitionContext partitionContext = partitionEvent.getPartitionContext();
        String contents = new String(event.getBody(), UTF_8);

        LastEnqueuedEventProperties properties = partitionEvent.getLastEnqueuedEventProperties();
        System.out.printf("Information received at %s. Last enqueued sequence number: %s%n",properties.getRetrievalTime(), properties.getSequenceNumber());

        System.out.printf("Partition[%s] with Offset-[%s] and Sequence Number[%s] has contents: '%s'%n",
                partitionContext.getPartitionId(),
                event.getOffset(),
                event.getSequenceNumber(),
                contents);
    };

    private final Consumer<Throwable> ERROR_HANDLER = errorContext -> {
        System.out.printf("Error occurred in partition processor");
        errorContext.printStackTrace();
    };


    public ClientSecretCredential createClientSecretCredential() {
        ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
                .clientId("****************")
                .clientSecret("******************")
                .tenantId("**********************")
                .build();
        return clientSecretCredential;
    }

I'm able to read all the events from a single subscription. However I need to do data analytics on these events from different subscriptions too. How do I configre Azure Eventhub to listen to multiple subscriptions ?

I read on Stackoverflow suggestions about creating consumer groups to solve this issue, however I'm not able to figure out how? I did create the consumer group, but how do i connect the newly created consumer group to different subscriptions in my azure aaccount and get the events streamed to the eventhub that i just created? [Note : I have followed - https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create to create an evenhub on azure]

Just in case if i need to clarify on what subscription I'm talking about, below is the screenshot

How do i achieve this? Thank you in advance

Maney

I'm not sure that I understand your goal. An Azure Subscription is the top-level container for resources and is a strong boundary. You cannot have an Event Hubs instance (or any other resource) that spans subscriptions. You can create an Event Hubs namespace in each subscription and then consume them from the same application - which may be what you're asking. The Event Hubs clients are scoped to a specific Event Hub; you'll need a different client for each that you'd like to read from. — Jesse Squire
Please edit your question to contain formatted text, not images of text (especially code). This meta post lists many reasons why this is important. — David Makogon
Also: just fyi - tool/product/service/tutorial/offsite resource recommendation questions are off-topic. — David Makogon
@DavidMakogon, Code formatted and I have edited my query. Thank you for pointing me to this :-) — Mahesh Maney

Mahesh Maney Mahesh Maney · Accepted Answer · 2020-12-26T17:58:38

So i figured out a way to solve my problem (stated above). After going through Microsoft documentation and some trial and error methods, Here's how i solved it;

I have SUBSCRIPTION-1 and SUBSCRIPTION-2. I have created an eventhub in SUBSCRIPTION-2. I go to SUBSCRIPTION-1 one and create a Resource-Group. After creating an Resource-Group, I create a EVENT-GRID. Within the eventgrid, I create a EVENT-SUBSCRIPTION that givens an option to point it to an endpoint. I chose the endpoint and selected the eventhub that was created in SUBSCRIPTION-1.

Now, i able to stream all the events from SUBSCRIPTION-1 to SUBSCRIPTION-2.

-Maney

Multiple Subscriptions in Azure Account Streaming to a single EventHub

1 Answers