AWS IoT scaling issues and metrics for monitoring IoT

Question

I am using AWS IoT Service.

When a device sends a registration message to MQTT broker, I have a rule to store it in a SQS queue. A Lambda function is triggered, when the message is added to the Queue. The Thing is created for the device and it's certificate is registered.

While carrying out the load testing, I observed that, after some time, the incoming messages are not received on the AWS MQTT broker and are not processed.

I have written some test clients which run on EC2 instances to simulate the MQTT clients.

If I restart the test clients after some time, again I can see the messages coming to AWS IoT.

I am not sure, if this is the issue of MQTT broker or if it is the issue with the clients running on EC2 instances.

I can think of possible issues because of limits on AWS IoT ,

https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_iot

I want to know what are the possible AWS IoT matrices, I need to monitor for this or which IoT specific alarms I need to configure?

Is it a possible issue on EC2 side? ( maybe network out bytes per second, etc.)

There is another load testing scenario, in which I am not doing registration of devices, but just capturing the connect or disconnect events. In this case, I am not observing similar issues.

chun geun yu chun geun yu · Accepted Answer · 2019-09-03T12:21:45

As you know, there is some limits about AWS IoT.

  API                  Transactions per Second
CreateCertificateFromCsr    15
CreateDynamicThingGroup 5
CreateJob   10
CreatePolicy    10
CreatePolicyVersion 10
CreateRoleAlias 10
CreateThing 15

Generally, AWS API throws Exception when it run over limts. How about catch Exception?

If you want to check EC2 network issue, use some command ( netstat, tcpdump, ... )

AWS IoT scaling issues and metrics for monitoring IoT

1 Answers