Grok filtering in logstash for multiple defined patterns

0
votes

I am trying to filter my logs matching few patterns I have. e.g:

E/vincinity/dholland_view_sql_global/IN/Cluster_Node/SSL-CACHE/Dsal1
F/vincinity/dholland_view_sql_local/IN/Cluster_Node3/SSL-CACHE/Dsal4

R/vincinity/dholland_view_sql_bran/IN/Cluster_Node/Sample/vr1.log

Now I want to grep these 3 paths from a bunch of logs: basically the pattern that I want to extract is logs containing "vincinity" "sql" and "IN" so with regex it would be simply *vincinity*sql*IN* I tried this grok filter:

grok {

    match => { "Vinc" => "%{URIPATHPARAM:*vincinity*sql*IN*}" }

  }

Then I get _grokparsefailure in kibana - I'm brand new to grok, so perhaps I'm not approaching this correctly.

2
elasticsearchlogstashlogstash-grok
Your expression is close. (?<field_name>.*?vincinity.*?sql.*?IN.*) would probably work well for you. The .*? construct is the non-greedy equivalent of .* and should make this perform somewhat better as a result. Less sub-string searching. – sysadmin1138

2 Answers

0
votes

From the grok filter documentation

The syntax for a grok pattern is %{SYNTAX:SEMANTIC}

The way the grok filter should work is

grok {
  match => {
    "message" => "%{PATTERN:named_capture}"
  }
}

Where message is the field that you want to parse, this is the default field that most inputs place your unparsed loglines in.

The URIPATHPARAM pattern is one predefined in logstash through a regex language called Onigurama. It may match your whole log message, but it will not capture certain chunks of it for you.

For help constructing a grok pattern, check out the docs, they link to a couple useful pattern construction tools.

0
votes

The correct format for using a custom pattern in your grok block is:

(?<field_name>the pattern here)

or you can define your own custom pattern (using regular expression) in seperate file (my-pattern.txt) like this :

MYPATH_MUST_BE_UPPERCASE Regex_Pattern

save it in ./patterns directory and then use it this way:

grok {
     patterns_dir => "./patterns"
     match => ["message" , "%{MYPATH_MUST_BE_UPPERCAS:path}"]
}

in your case : 

(?<vincinity>(?>/\s*.*?vincinity.*?\s*)+)
(?<sql>(?>/\s*.*?sql.*?/\s*)+)
(?<in>(?>\s*.*?(IN).*?\s*)+)