Filtering Bluemix cloud foundry ERR logs on logstash

Question

I am currently working on setting up ELK stack on Bluemix containers. By following this blog, I was able to create a logstash Drain and get all the Cloud Foundry logs from the Bluemix web app into logstash.

Is there a way to filter out logs based on log levels? I am trying to filter out ERR in logstash output and send them to Slack.

The following code is the filter configuration of the logstash.conf file:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|) +%{GREEDYDATA:syslog5424_msg}" }
    }

I am trying to add a Slack webhook to the logstash.conf output so that when a log level with ERR is detected, the error message is posted into the Slack channel.

My output conf file with the Slack HTTP post looks something like this code:

output {

 if [loglevel] == "ERR" {
        http {
            http_method => "post"
            url => "https://hooks.slack.com/services/<id>"
            format => "json"
            mapping => {
                "channel" => "#logstash-staging"
                "username" => "pca_elk"
                "text" => "%{message}"
                "icon_emoji" => ":warning:"
            }
        }
    }

    elasticsearch { }
}

Sample Logs from cloud Foundry:

2016-05-25T13:14:51.269-0400[App/0]ERR npm ERR! There is likely additional logging output above.
2016-05-25T13:14:51.269-0400[App/0]ERR npm ERR! npm owner ls pca-uiapi
2016-05-25T13:14:51.274-0400[App/0]ERR npm ERR! /home/vcap/app/npm-debug.log
2016-05-25T13:14:51.274-0400[App/0]ERR npm ERR! Please include the following file with any support request:
2016-05-25T13:14:51.431-0400[API/1]OUT App instance exited with guid cc73db5d-   6e8c-4ff4-b20f-a69d7c2ba9f4 payload: {"cc_partition"=>"default", "droplet"=>"cc73db5d-6e8c-4ff4-b20f-a69d7c2ba9f4", "version"=>"f9fb3e09-f234-43d4-94b1-a337f8ad72ad", "instance"=>"9d7ad0585b824fa196a2a64e78df9eef", "index"=>0, "reason"=>"CRASHED", "exit_status"=>1, "exit_description"=>"app instance exited", "crash_timestamp"=>1464196491}
2016-05-25T13:16:10.948-0400[DEA/50]OUT Starting app instance (index 0) with guid cc73db5d-6e8c-4ff4-b20f-a69d7c2ba9f4
2016-05-25T13:16:36.032-0400[App/0]OUT > [email protected] start /home/vcap/app
2016-05-25T13:16:36.032-0400[App/0]OUT > node server.js
2016-05-25T13:16:36.032-0400[App/0]OUT
2016-05-25T13:16:37.188-0400[App/0]OUT PCA REST Service is listenning on port: 62067
2016-05-25T13:19:02.241-0400[App/0]ERR at Layer.handle_error (/home/vcap/app/node_modules/express/lib/router/layer.js:71:5)
2016-05-25T13:19:02.241-0400[App/0]ERR at /home/vcap/app/node_modules/body-parser/lib/read.js:125:7
2016-05-25T13:19:02.241-0400[App/0]ERR at Object.module.exports.log (/home/vcap/app/utils/Logger.js:35:25)

Is there a way to get this working? Is there a way to check the log level of each message? I am kinda stuck and was wondering if you could help me out.

In the Bluemix UI, the logs can be filtered based on the channel ERR or OUT. I could not figure how to do the same on logstash.

Thank you for looking into this problem.

@Val : I have added some of the logs above, please take a look. — Lokesh Sreedhar

Val Val · Accepted Answer · 2016-05-26T03:21:30

The grok provided in that article is meant to parse the syslog message coming on port 5000. After all syslog filters have run, your application log (i.e. the sample log lines you've shown in your question) are in the @message field.

So you need another grok in order to parse that message. So after the last mutate you can add this:

grok {
    match => {"@message" => "%{TIMESTAMP_ISO8601:timestamp}\[%{WORD:app}/%{NUMBER:num}\]%{WORD:loglevel} %{GREEDYDATA:log}"}
}

After this filter runs, you'll have a field named loglevel which will contain either ERR or OUT and in the former case will activate your slack output.

Filtering Bluemix cloud foundry ERR logs on logstash

1 Answers