Azure Stream Analytics 'TimeStamp By' in query doesn't works on job but works fine on test

Question

I'm working on a IoT project. I've a Raspberry pi which send data to an IoTHub on Azure. That Hub forwards that data to an Azure Stream Analytics Job. In my query I try to aggregate(here an average) all the data in a TumblingWindow of 1 minute but as a Timestamp I use a custom datetime sent in the data.

I've tried many things but nothing seemed to work. the stream job seems to ignore the datetime I provide and just aggregate everything based on the arrival time. Only when using the "Test" functionality by uploading a json file does it works.

SELECT
    DateAdd(minute, -1, system.Timestamp) as TumblingWindowStartTime, 
    system.TimeStamp as TumblingWindowEndTime, 
    event.DeviceId as DeviceId, 
    AVG(event.Temperature) as TemperatureAverage,
    Count(*) as MeasurementsCount
INTO
    [input]
FROM
    [output] as event
TIMESTAMP BY MeasuredOn
GROUP BY
    event.DeviceId,
    TumblingWindow(minute, 1)

Type of data I send

[
   {
      "Temperature":13.426585352712585,
      "DeviceId":"UlyssesPi",
      "MeasuredOn":"2019-04-09T11:20:30.1027311Z"
   },
   {
      "Temperature":16.81523611620778,
      "DeviceId":"UlyssesPi",
      "MeasuredOn":"2019-04-09T11:20:35.2281002Z"
   },
   ...
]

What I'm expecting is what I get when using the "Test" functionality on Azure Stream Analytics :

TUMBLINGWINDOWSTARTTIME || TUMBLINGWINDOWENDTIME || DEVICEID || TEMPERATUREAVERAGE || MEASUREMENTSCOUNT
"2019-04-09T11:20:0...     "2019-04-09T11:21:0...   UlyssesPi   14.674093214798454    6
"2019-04-09T11:21:0...     "2019-04-09T11:22:0...   UlyssesPi   18.612186615873217    12
"2019-04-09T11:22:0...     "2019-04-09T11:23:0...   UlyssesPi   12.799415359568199    12

but instead I get the following:

TUMBLINGWINDOWSTARTTIME || TUMBLINGWINDOWENDTIME || DEVICEID || TEMPERATUREAVERAGE || MEASUREMENTSCOUNT
"2019-04-09 11:22:0...     "2019-04-09 11:23:0...   UlyssesPi   15,4994594331363      30

So how come the results aren't the same? Is it something I did wrong? What can I do?

Thanks for your help.

Sid Ramadoss Sid Ramadoss · Accepted Answer · 2019-04-11T17:53:18

When testing your Stream Analytics logic on portal, event ordering policies don't get applied. However, when your running the job on cloud, the event ordering policies do get applied. You need to ensure the timestamps in your payload is accurate (and event aren't arriving very late). You can also configure the event ordering policies to define late arrival policies (default is 5 seconds). You can learn how to use those settings here.

Azure Stream Analytics 'TimeStamp By' in query doesn't works on job but works fine on test

1 Answers