Azure streaming analytics with event hub input stream position

Question

Setup

I use Azure stream analytics to stream data into Azure warehouse staging table. The input source of the job is a EventHub stream. I notice when I'm updating the job, the job input event backlog goes up massively after the start. It looks like the job starting to process the complete EventHub queue again from the beginning.

Questions

• how is the stream position management organised in stream analytics
• is it possible to define a stream position where the job starts (event after queued after a specific point in time for example)

So far done

I notice a similar question here on StackOverflow. There is mentioned a variable name "eventStartTime". But since I use an "asaproj" project within visual studio to create, update and deploy the job I don't know where to place this before deploying.

Answer 1

For updating job without stop, it will use previous setting of "Joboutputstarttime", so it is possible for job starting to process the data from the beginning.

you can stop the job first, then choose "Joboutputstarttime" before you will start the job.

You can reference this document https://docs.microsoft.com/en-us/azure/stream-analytics/start-job to see detailed information for each mode. for your scenario, "When last stopped" mode maybe the one you need and it will not process data from beginning of the eventhub queue.