The closest thing to an "out of the box" feature for this would be to use IoT Hub queries. You can use device twins to set the device status. The status could be "nominal/alarm", that way you can use the IoT Hub Query language to extract a list of devices currently in a certain state.
You would need to update your devices to have the device report updates to the device twin, or you need to write a piece of code that takes the messages and updates the twin from the backend.
You mentioned a dashboard, I'm not aware of any out of the box solution for this, but you could use PowerBI to create a dashboard based on a number of data sources. There are many Azure components you can store your telemetry in, and you can use PowerBI to create a list of devices whose last message was an alarm. Putting Stream Analytics in between would enable you to store just the alarms in a database.
In short: there are many ways of getting to the same result, as far as I know, none will be out of the box.