Azure Stream Analytics job truncates data

Question

I'm having quite a strange issue with my Azure IoT Hub and Azure Stream Analytics job.

Long story short: I configured an Azure IoT Hub instance to get some weather data from a Raspberry PI 3 which had a sensor attached to it; then, I configured an Azure Stream Analytics job to take the data from the IoT hub and put it into a DocumentDB collection - during the configuration phase, I set the sensorId as being the partition key. After running the job for almost 3 days, I bought some more sensors (that have different IDs) and add them to the Raspberry PI - the data is being collected and sent to the Azure IoT Hub without any issues for all 3 attached sensors (checked the transmission with the Device Explorer).

Now comes the weird part - only the data sent by first sensor (the one which run for almost 3 days by itself) is getting through the Analytics job and sent to DocumentDB - in the portal I see that I have a lot of Input events but I see 0 Output events (though data is getting through).

In order to exclude any other issues, I wrote a small console application that "listens" to the IoT hub for the incoming messages and indeed, all sensors are sending data to the IoT hub.

The query used for the analytics job is the following:

    SELECT
    sensorId,
    avg(humidity) as avghumidity,
    avg(temperatureFromHumidity) as avgtemperatureFromHumidity,
    avg(objectTemperature) as avgobjectTemperature,
    avg(temperatureFromIr) as avgtemperatureFromIr,
    avg(pressure) as avgpressure,
    avg(lux) as avglux,
    System.TimeStamp AS executionTime
INTO
    [XXX-document-db]
FROM
    [home-meteo] TIMESTAMP BY currentTime
GROUP BY 
    sensorId,
    TumblingWindow(second, 30)

I've seen other issues somehow similar but the accepted answer was that there was a bug at Microsoft.

There is any way on how to debug this thing?

Thank you.

Though I had some suspicions, I said that is a good idea to try this: I stopped the analytics job, added a new output (this time as a table storage, though I don't think it matters), I modified the query to send the data to the new output and then run the job again - I have data!!!! Maybe this is a bug? — Edi
If nothing else changed, just the output, then it sounds like a problem with the Azure Streaming output. I will ask the ASA team to investigate and close the loop on this. — Ryan CrawCour

Edi Edi · Accepted Answer · 2016-03-17T20:08:18

My bad.... Accidentally I found the issue - in the documentdb I was seeing the records only from one sensor but not from the other 2 and didn't understood why but after looking at the query that I wrote in the analytics console, I realized that was only my fault.

The scenario was like this: I configured the job output for the documentdb to have the executionTime field as row key but this value was identical for all 3 sensors and I imagine that after the first insert into the DocumentDB, the other two gave some sort of an error (I'm not familiar with DocumentDB - I know SQL Server) and this is what I think happened.

I change the query and took another field as row key and everything went fine. Hopefully I'm clear enough.

Azure Stream Analytics job truncates data

2 Answers