How do I monitor AWS IOT connected devices history and group by shadow state/attribute

Question

Given I have xK Devices, Each of them connect to AWS IoT and each of them have attributes/shadow states to group them

device shadow example: { "factory": "factoryA", "pipeInstalledVersion: "1.5.6" }

What's the easiest way to monitor these devices on a grouped basis (Based on the shadow state)

E.g. I want to know how many devices are connected in factory A at 11:05, 15:30, 20:50. I also want to know what pipeInstalledVersion the devices have at a particular time interval (e.g. every 5 minutes). I also want to know e.g. if there are less than X devices connected at 09:00 at factory A, then send an alert.

AWS IoT has a monitoring dashboard for connected devices but there is no way to group that by shadow state/attribute

I've looked into AWS IoT analytics but it looked like there were some limitations - The recommended platform QuickSight to visualise data has a limited auto refresh period (1 hour I believe), even though the underlying dataset can be refreshed every ~5 minutes. - The dataset will only show data if an IOT device has transmitted data in that time. What if the IOT device is connected but it does not transmit data in that time period? It will be as if it is not connected.

Answer 1

Fleet indexing provides the capability to powerfully search over your fleet of devices using Thing attributes and Thing Shadow state.

Combine this with AWS Lambda and you have yourself scheduled searches over your fleet that can be paired with any number of AWS actions (i.e. record CloudWatch metric, scale EC2, and so on).

Sample Fleet indexing queries:

connectivity.connected:true

returns all Things currently connected.

connectivity.connected:true AND shadow.reported.model:A 

returns all Things currently connected and have a particular Shadow state.

aws iot get-cardinality --aggregation-field "connectivity.connected" --query-string "*"

To find # of connected devices at a given time. (aggregation queries)