Runtime errors for Azure stream Analytics Job

Question

I tried to modify an existing azure stream analytics job by adding one more temporary result set. But when I run the SA job, it is throwing runtime error and watermark delay is getting increased.

Below is the existing SAQL in the Stream Analytics job:

-- Reading from Event Hub
WITH INPUTDATASET AS (
    SELECT 
        udf.udf01([signals2]) AS flat
    FROM [signals2]
    PARTITION BY PartitionId
    WHERE [signals2].ABC IS NOT NULL
),

INPUT1 AS (
    SELECT
        ID,
        SIG1,
        SIG2
    FROM [signals2] as input
    WHERE GetArrayLength(input.XYZ) >=1
)

--Dump the data from above result set into cosmosDB

I tried to add the below temporary result set to the SAQL:

INPUT2 AS (
    SELECT
        ID,
        SIG3,
        SIG4
    FROM [signals2] as input
    WHERE GetArrayLength(input.XYZ) =0
)

Now when I start the SA job, it throws runtime error.

When I fetch the logs below is the error logs i received.

TimeGenerated,Resource,"Region_s",OperationName,"properties_s",Level
"2020-01-01T01:10:10.085Z",SAJOB01,"Japan West","Diagnostic: Diagnostic Error","{""Error"":null,""Message"":""First Occurred: 01\/01\/2020 01:10:10 | Resource Name: signals2 | Message: Maximum Event Hub receivers exceeded. Only 5 receivers per partition are allowed.\r\nPlease use dedicated consumer group(s) for this input. If there are multiple queries using same input, share your input using WITH clause. \r\n "",""Type"":""DiagnosticMessage"",""Correlation ID"":""xxxx""}",Error
"2020-01-01T01:10:10.754Z",SAJOB01,"Japan West","Receive Events: ","{""Error"":null,""Message"":""We cannot connect to Event Hub partition [25] because the maximum number of allowed receivers per partition in a consumer group has been reached. Ensure that other Stream Analytics jobs or Service Bus Explorer are not using the same consumer group. The following information may be helpful in identifying the connected receivers: Exceeded the maximum number of allowed receivers per partition in a consumer group which is 5. List of connected receivers - AzureStreamAnalytics_xxxx_25, AzureStreamAnalytics_xxxx_25, AzureStreamAnalytics_zzz_25, AzureStreamAnalytics_xxx_25, AzureStreamAnalytics_xxx_25. TrackingId:xxx_B7S2, SystemTracker:eventhub001-ns:eventhub:ehub01~26|consumergrp01, Timestamp:2020-01-01T01:10:10 Reference:xxx, TrackingId:xxx_B7S2, SystemTracker:eventhub001-ns:eventhub:ehub01~26|consumergrp01, Timestamp:2020-01-01T01:10:10, referenceId: xxx_B7S2"",""Type"":""EventHubBasedInputQuotaExceededError"",""Correlation ID"":""xxxx""}",Error...

For the SA job, the input signals2 is having a dedicated consumer group (consumergrp01)

For this Stream Analytics job, dedicated consumer group is available.There are 3 readers on a partition for this consumer group, but still it is throwing the error as Maximum Event Hub receivers exceeds. Why is it so?

Jay Gong Jay Gong · Accepted Answer · 2020-01-03T08:06:58

Message: Maximum Event Hub receivers exceeded. Only 5 receivers per partition are allowed.

I think the error message is clear on the root cause of the runtime error.Please refer to the statement in this doc:

There can be at most 5 concurrent readers on a partition per consumer group; however it is recommended that there is only one active receiver on a partition per consumer group. Within a single partition, each reader receives all of the messages. If you have multiple readers on the same partition, then you process duplicate messages. You need to handle this in your code, which may not be trivial. However, it's a valid approach in some scenarios

You could follow the suggestion in the error message:If there are multiple queries using same input, share your input using WITH clause. Try to isolate input2 as a temporary result set then use it in other temp results.

Runtime errors for Azure stream Analytics Job

1 Answers