Azure IoT Hub - Recovering from a device flooding the hub with reported device twin messages

0
votes

I have an Azure IoT Hub application and a device just started to send messages indicating changes in reported device twin properties every other second. In a matter of few hours, the total messages that day went over 50k. When this number goes of 40k, the IoT Hub becomes VERY slow to respond for ALL customers - not only until the device is shut off but until all those messages have managed to throttle through the system which seems to be after several hours or until next morning.

So if this type of flooding happens, the entire system for all customers grinds to a halt due to slowness.

This is a device bug and needs to be fixed but I was wondering if there is a way - IF this happens - to get the whole IoT hub back to normal where it isn't slow? Something like kicking the offending device or rebooting the hub or something. Or better yet - is there a way to prevent devices from flooding the hub faster than an x amount of messages per minute or something?

2
azureazure-iot-hub
What's the scale tier and number of units used for IoT Hub? – Roman Kiss
It's S1 - Standard. Also - Number of IoT Hub units : 1 – Jon Th
Try to switch to the S3, which its throttling limit is ~5x of the S1. Note, that the price is the highest, such as $2,500 per unit – Roman Kiss
Yes, I thought of that, but that's not really a solution. That's like buying more hardware for more speed. I need to address the root of the problem. Devices shouldn't be able to send messages that fast or I should be able to get the overall speed of the IoT Hub back to normal for all customers very quickly. – Jon Th
May be the Failover feature can help you here, see more details docs.microsoft.com/en-us/azure/iot-hub/tutorial-manual-failover, – Roman Kiss

2 Answers

0
votes

You could build some logic to kick the offending device when it starts spamming your hub. One approach might be to route all twinChangeEvents to a separate endpoint and write a Stream Analytics Job to group the messages per deviceId and keep a count of the events in a sliding window of X minutes. After the count reaches a threshold you set, you could call an Azure Function to disable the device and send a notification.

There is one caveat, the docs state:

If the rate of change is too high, or for other reasons such as internal failures, the IoT Hub might send only one notification that contains all changes.

I don't know if your device reaches that rate, but I think this would be a suitable approach to kick the offending device.

0
votes

At some point the IoT hub will start rejecting your messages once the throttling limit is reached as per your tier and units purchased. Now, to handle excess of messages either throttling reached or the IoT hub is slow in processing the message you should auto-scale your IoT hub.

As per the docs in above link provided:-

The sample solution outlined in this article provides the ability to monitor an IoT Hub for the case where the current message count has exceeded a set threshold (for example, 90% of the allowed messages) and, in that case, to automatically scale the IoT Hub up to the next unit of capacity.

At the end of the day you need to also auto-down scale your IoT hub so that at low traffic received the cost is not high for IoT hub. Check the Scaling down section in the article link above.