Stackdriver alert policy not triggering

Question

I have a custom logs based metric, based on logs emitted by my application, like this:

resource.type="k8s_container" resource.labels.cluster_name="my-cluster" resource.labels.namespace_name="preprod" resource.labels.container_name="my-app" jsonPayload.logType="MyEventType"

I wanted to setup a custom alerting policy based on it and make it trigger an alert if there's no logs for 10mins.

I did something like the following:

{
  "combiner": "OR",
  "conditions": [
    {
      "conditionAbsent": {
        "aggregations": [
          {
            "alignmentPeriod": "60s",
            "perSeriesAligner": "ALIGN_SUM"
          }
        ],
        "duration": "600s",
        "filter": "metric.type=\"logging.googleapis.com/user/my-metric-preprod\" resource.type=\"k8s_container\"",
        "trigger": {
          "percent": 100
        }
      },
      "displayName": "No log events for 10 mins"
    }
  ],
  "displayName": "My custom policy - preprod",
  "enabled": true,
  "notificationChannels": // doesn't matter
}

Although everything looks fine, when I stop emitting events, nothing is executed.

Am I missing something here?

Adad O. Adad O. · Accepted Answer · 2019-08-26T21:37:40

I've replicated this with the same values in your JSON;

   {
  "combiner": "OR",
  "conditions": [
    {
      "conditionAbsent": {
        "aggregations": [
          {
            "alignmentPeriod": "60s",
            "perSeriesAligner": "ALIGN_SUM"
          }
        ],
        "duration": "180s",
        "filter": "metric.type=\"logging.googleapis.com/user/test-alerting\" resource.type=\"k8s_container\"",
        "trigger": {
          "percent": 100
        }
      },
      "displayName": "logging/user/test-alerting"
    }
  ],
  "displayName": "testing",
  "enabled": true,
  "notificationChannels": [
    "NOTIFICATION/CHANNEL/123"
  ]
}

Not sure why you're alerting is not working, have you verified the notification Channel?

Stackdriver alert policy not triggering

1 Answers