Sensu remediation does not work

Question

I have configured the following check:

    "cron": {
    "command": "check-process.rb -p cron",
    "subscribers": [],
    "handlers": [
        "mailer",
        "flowdock",
        "remediator"],
    "interval": 10,
    "occurences": 3,
    "refresh": 600,
    "standalone": false,
    "remediation": {
       "light_remediation": {
         "occurrences": [1, 2],
         "severities": [2]
    }
 }
},
  "light_remediation": {
  "command": "touch /tmp/test",
  "subscribers": [],
  "handlers": ["flowdock"],
  "publish": false,
  "interval": 10
},

Mailer and flowdock handlers are being executed as expected, so I am receiving e-mails and flowdock notifications when cron service is not running. The problem is that remediator check is not working and I have no idea why. I have used this: https://github.com/nstielau/sensu-community-plugins/blob/remediation/handlers/remediation/sensu.rb

Anthony Hobbs Anthony Hobbs · Accepted Answer · 2017-02-13T15:25:10

I ran into similar issues but finally managed to get it working with some modifications.

First off, the gotchas:

Each server (client.json.template) needs to subscribe to a channel $HOSTNAME

  "subscribers": ["$HOSTNAME"],

You don't have a "trigger_on" section, which is in the code but not the example and you want to set that up to trigger on the $HOSTNAME as well. my_chek.json.template

  "trigger_on": ["$HOSTNAME"]

The remediation checks need to subscribe to $HOSTNAME as well (so you need to template the checks out as well)

  "subscribers": ["$HOSTNAME"],

At this point, you should be able to trigger your remediation from the sensu server manually.

Lastly, the example code listed in sensu.rb is broken... The occurrences check needs to be up one level in the loop, and the trigger_on is not inside the remediations section, it's outside.

    subscribers = @event['check']['trigger_on'] ? [@event['check']['trigger_on']].flatten : [client]

...

        # Check remediations matching the current severity
        next unless (conditions["severities"] || []).include?(severity)

        remediations_to_trigger << check
      end
   end
   remediations_to_trigger
end

After that, it should work for you.

Oh, and one last gotcha. In your client.json.template

"safe_mode": true

It defaults to false...

Sensu remediation does not work

1 Answers