bosun never send unknown notifications

Question

Unknown notfication is really struggling me and the official docs seems newer than released bosun binary, like unknownPost is ilegal option.

The following unknown template and notification do not work. I can see unknown notfications in bosun UI, but it doesn't send notification(http alert). Other normal alerts work fine.

template tmpl_unknown {
    body = 'x'
    subject = `{{$v := .Eval .Alert.Vars.val }}{"warn_undo": [{"app":"xxx","db_type":"NOSQL","db_cat":"aaaa","ip":"{{.Group.host}}","warn_level":1,"warn_type":"cpu","warn_detail":"cpu {{.Group.device}} total util is {{$v.Value | pct}} ","notifier": "xxxxxxxx","status":"undo","report_time":""}]}`

    # body = {{.Name}}: {{.Group | len}} unknown alerts
    # subject = `{"warn_undo": [{"app":"aaaa","db_type":"NOSQL","db_cat":"aaaa","ip":"x.x.x.x","warn_level":1,"warn_type":"alive","warn_detail":"Time: {{.Time}} xName: {{.Name}}  Alerts:{{range .Group}}{{.}},{{end}}","notifier": "xxxxxxxx","status":"undo","report_time":""}]}`

    unknown_alive = `
    {"warn_undo": [{"app":"aaaa","db_type":"NOSQL","db_cat":"aaaa","ip":"x.x.x.x","warn_level":1,"warn_type":"alive","warn_detail":"Time: {{.Time}} Name: {{.Name}}  Alerts:{{range .Group}}{{.}},{{end}}","notifier": "xxxxxxxx","status":"undo","report_time":""}]}`
}

unknownTemplate = tmpl_unknown


notification gcsx_unknown {
        post = ${sys.gcsAlert}
        contentType = application/json
        print = True
        timeout = 5m
        next = gcsx_unknown
        unknownBody = unknown_alive
        bodyTemplate = subject
        # unknownMinGroupSize = 5
        # unknownThreshold = 0
}

alert cpu_total_util_or_unknown {
        template = tmpl_unknown
        $val = avg(q("none:0all-last:cpu.pct_used{host=*,device=total}", "1m", ""))
        warn = $val > 60
        warnNotification = gcsx_unknown
        runEvery = 1
        unknown = 1m
}

I want to alert cpu usage. If host is gone, no data reported and I will got an unknown alert.

Bosun version: the newest compiled from master branch(2018.10.08)

seanlook seanlook · Accepted Answer · 2018-10-11T08:27:45

After digging the code, I found the issue finally and it's simply:

unknown notification must be sent as critNotification

cmd/bosun/sched/check.go:290

    incident.NeedAck = true
    switch event.Status {
    case models.StCritical, models.StUnknown:
        notify(a.CritNotification)
    case models.StWarning:
        notify(a.WarnNotification)
    }

Change warn, warnNotification to crit and critNotification and notfication is sent successfully. This should be documented in bosun.org.

unknownPost is a legal option, just like unknownPost,unknownGet,unknownBody,unknownEmailSubject. But they should be defined in template variables, or it will report error:

2018/10/11 16:19:36 fatal: main.go:130: couldn't read rules: conf: bosun_rule.conf:47:0: at <alert x...>: 
notification x uses template key unknown_post in unknown post url, but template cpu_util does not include it

The right way is:

template tmpl_unknown {
    unknown_alive = `
    {"warn_undo": [{"app":"xxx","db_type":"NOSQL","db_cat":"aaaa","ip":"x.x.x.x","warn_level":1,"warn_type":"alive","warn_detail":"Time: {{.Time}} Name: {{.Name}}  Alerts:{{range .Group}}{{.}},{{end}}","notifier": "xxx","status":"undo","report_time":""}]}`
    unknown_post = "//x.x.x.x:7777/alert"
    # unknown_post = {{ V "$gcsAlert" }}
}

notification gcsx_unknown {
        post = ${sys.gcsAlert}
        contentType = application/json
        print = True
        timeout = 5m
        next = gcsx_unknown
        unknownBody = unknown_alive
        unknownPost = unknown_post
}

As you note that the http url prefix http: need to be removed.

bosun never send unknown notifications

1 Answers