Filtering specific lines from log file in logstash

Question

I am not able to get specific lines from logs file /var/log/messages. I am using logstash-forwarder in client-server and logstash, elasticsearch and kibana in log-server. I tried to install grep filter but it gives me some error so I try to implement below with grok. My original post is here . I found this but m quite unsatisfied.

Following is the configuration for logstash-forwarder file-name: logstash-forwarder in client-server

{
  "network": {
    "servers": [ "logstashserver-ip:5000" ],
    "timeout": 15,
    "ssl ca": "xxx.crt"
  },
  "files": [
    {
      "paths": [
        "/var/log/messages"
       ],
      "fields": { "type": "syslog" }
    }
   ]
}

and following is the logstash configuration in logstashserver

file-name:input.conf

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "xxx.crt"
    ssl_key => "xxx.key"
  }
}

file-name:filter.conf

filter {
    grok {
        match => ["message", "\[%{WORD:messagetype}\]: %{GREEDYDATA}"]
    }
}

file-name:output.conf

output {
  elasticsearch { host => "logstashserver-ip" }
  if [messagetype] == "ERROR" {
        stdout {
            codec => "rubydebug"
        }
}
}

Is there anything wrong?

Could you describe in more detail which messages you'd like to drop? You seem to understand conditionals, so I'm not sure what the problem is. If you don't want certain messages making it to ES, you shouldn't be sending them there... — rutter
I only like to view WARNING and ERROR log in kibana from /var/log/messages and same goes for /var/log/secure. In this case I am only concern with /var/log/messages first. — user3884162
Question: Do you want "ERROR" message types to reach elasticsearch only, and not others? — DBPriGuy

SimonH SimonH · Accepted Answer · 2015-03-17T17:02:40

Not sure if you're still having this problem, but I'd look at dropping the messages you don't want. On my server, I get syslog severity levels which include syslog_severity_code as defined at http://en.wikipedia.org/wiki/Syslog#Severity_levels.

If you're getting them in your indices, try something like

filter {
    if [type] == 'syslog' and [syslog_severity_code] > 5  {
        drop { }
    }
}

Filtering specific lines from log file in logstash

1 Answers