There are a couple of ways you can achieve this.
The best way is probably to configure notifications in Pacemaker. Pacemaker has support for SNMP, SMTP, and custom notification agents built in. The Pacemaker documentation is a good place to start researching: http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Pacemaker_Explained/ch07.html
However, as an example, if you wanted to write your own notification agent, you could create a script somewhere the cluster can access, like /usr/local/bin/notification-script-example.sh, with something like this:
#!/bin/bash
### list of variables passed from Pacemaker to all alert agents:
# CRM_notify_recipient the configure recipient
# CRM_notify_node name of affected node
# CRM_notify_desc details about event
# CRM_notify_task requested fencing/resource opertion (fencing and resource notify only)
# CRM_notify_rc return code from operation (fencing and resource notify only)
# CRM_notify_rsc name of the affected resource (resource notify only)
# CRM_notify_target_rc expected return code of an operation (resource notify only)
# CRM_notify_status numerical code used by Pacemaker to represent operation result (resource notify only)
### some variables won't get values; set NA if not populated:
CRM_notify_recipient="${CRM_notify_recipient:-NA}"
CRM_notify_node="${CRM_notify_node:-NA}"
CRM_notify_desc="${CRM_notify_desc:-NA}"
CRM_notify_task="${CRM_notify_task:-NA}"
CRM_notify_rc="${CRM_notify_rc:-NA}"
CRM_notify_rsc="${CRM_notify_rsc:-NA}"
CRM_notify_target_rc="${CRM_notify_target_rc:-NA}"
CRM_notify_status="${CRM_notify_status:-NA}"
### do something with these values
logger "PCMK-NOTIFY: recipient: $CRM_notify_recipient"
logger "PCMK-NOTIFY: affected node: $CRM_notify_node"
logger "PCMK-NOTIFY: event details: $CRM_notify_desc"
logger "PCMK-NOTIFY: requested op: $CRM_notify_task"
logger "PCMK-NOTIFY: op ret code: $CRM_notify_rc"
logger "PCMK-NOTIFY: affected res: $CRM_notify_rsc"
logger "PCMK-NOTIFY: expected rc: $CRM_notify_target_rc"
logger "PCMK-NOTIFY: pcmk result: $CRM_notify_status"
### exit
exit 0
Then you would configure the notification agent inside of pacemaker as a primitive with the ClusterMon resource-agent, then clone it so it runs on all nodes (in crmsh syntax):
primitive cluster-notifications ocf:pacemaker:ClusterMon \
params user=root update=30 extra_options="--watch-fencing \
-E /usr/local/bin/notification-agent-example.sh"
clone cl-cluster-notifications cluster-notifications
For your script, it sounds like you'd want to send out an email, so you'd just need to alter the ### do something with these values section of the script to send everything in an email rather than simply logging.
Another way to get notifications of events in Pacemaker, could be to create a MailTo primitive in your cluster using the ocf:heartbeat:MailTo resource-agent. It will send an email anytime it is started or stopped in the cluster, so usually you would want to order it to start after everything else in the cluster; that will ensure it starts last and stops first if a migration/failover happens.
If you're only interested in DRBD notifications, you could use some of the handler scripts that ship with DRBD by enabling them in the handler{} section of /etc/drbd.d/global_common.conf, or you could add the following to the resource configurations you care about:
handlers {
split-brain "/usr/lib/drbd/notify-split-brain.sh [email protected]";
out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh [email protected]";
}
Obviously, there are many ways to skin the proverbial cat, but I hope that information is helpful!
