Stackdriver Alerts for “Decreases By” Condition Misfiring

Question

I have a custom metric being logged to Stackdriver from a service running in GKE. This custom metric is sort of a load factor for the service. I want to get notified if this load drops by 10% over 5 minutes. This seems pretty straghtforward via the UI:

However, when I set up this alert I begin receiving endless notifications such as:

ALRT [alert name] on [project-name] decreasing by -0.116%
ALRT [alert name] on [project-name] decreasing by 0.207%
...

One alert for each trend line as configured, but each "decreases by" level is well below the 10% threshold that I've set. I have 26 instances of this service and this is resulting in my phone getting blown up with texts every few minutes.

I've also tried setting up conditions for individual series in this metric and the same error occurs: alerts are sent when the change is < 10%

What's the correct method for configuring the "decreases by" condition?

Answer 1

try to change the Condition triggers if to Percent of time series violates. I have no clue, but would assume that else Number of time series violates might also trigger the alert.