Send one or more events to EventHub Azure

0
votes

I am using an Logic App (LA) on Azure to query my db every 3 mins. Then the LA uses an EventHub connector to send my query result, the table, to Azure Stream Analytics (ASA).

Normally the result table has around 100 rows, definitely many more in peak time.

I thought sending Eventhub message one row each time, would incur so many calls, hence perhaps delay the ASA's logic(?)

My questions are:

  • How to send multiple messages thru the LA's Eventhub Action Connector? I see there's one option: Send one or more events to Eventhub, but wasn't able to figure out what to put in the content. Tried putting the table(the array). The following request body works. e.g body:

    [ { "ContentData": "dHhuX2FnZV9yZXN1bHQ=", "Properties": { "tti_IngestTime": "2018-09-26T20:10:55.4480047+00:00", "tti_SLAThresholdMins": 330, "MinsPastSla": -6 } }, { "ContentData": "AhuBA2FnZV9yZXN1bHQ=", "Properties": { "tti_IngestTime": "2018-09-26T20:10:55.4480047+00:00", "tti_SLAThresholdMins": 230, "MinsPastSla": -5 } } ]

    • Sending 100 events one by one to ASA, is there any performance concern?

Thank you!

1
azureazure-logic-appsazure-eventhub
Regarding is there any performance concern? That depends on your performance requirements. I've been sending thousands of events one by one from a Logic App to an Event Hub (with some additional processing included), which took a maximum of maybe very few minutes. But whether this is good or bad is up to your use case. Also numbers maybe totally different for you. – Onkel Toob
Thank you for the reply! I don't seem to find much of the ASA message schema instruction. So I looked at my one-by-one messages to eventHub, and mimic that to compose the multiple message. (Now updated in the question body) So basically, single message is { "ContentData": someBase64String, "Properties": { ... my message object ...} } , then I create an array that contains multiple msg. Still cannot sample from ASA. :( – ayuspark
I've never tried to send multiple messages to EH within one request. It should of course be possible to wrap these messages in a valid JSON object, however, that would always be treated as one event by ASA. That's probably not what you want. Have you tried sending each event one by one? – Onkel Toob
Thank you again, Onkel Toob, I have randomly discovered the seemingly workable answer. I posted below. Wow, that didn't take long at all... T_T – ayuspark

1 Answers

0
votes

Seem to find the answer.

(1) the JSON I am sending looks correct, and the post request to EvenHub is successful.

Post body is [{}, {}, {}], which is the correct format

(2) ASA couldn't read the stream is likely due to not able to deserialize the messages from EventHub.

I happen to change how I encode the base64 string for the "ContentData" send to the EventHub. It looks like the message sent to the EH,

{ "ContentData": "some base64() string", "Properties": {} },

the base64() needs to encode the "Properties" value, not anything else, for ASA to be able to deserialize the message.

It didn't work because I encoded using a random string instead of the value of the "Properties".