Grok pattern for BIND9 log

Question

I need to write a grok filter for bind9 DNS logs. A sample log looks like this:

17-Feb-2018 23:06:56.326 queries: info: client @0x563d72c3ea20 172.26.0.1#34564 (test.example.com): query: test.example.com IN A +E(0)K (172.26.0.3)

I validated the following pattern on grokconstructor, where it successfully matches the log above:

filter {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:logdate} queries: info: client @0x.{16} %{IP:source_ip}#(?<source_port>[0-9]+) \(%{HOSTNAME:query}\): query: .*$" }
    }
    date {
        match => ["logdate", "dd-MMM-yyyy HH:mm:ss.SSS"]
    }
}

However on Kibana my log is tagged with _grokparsefailure and is not parsed.

I don't understand how your pattern could have been validated. TIMESTAMP_ISO8601 can't match 17-Feb-2018 since it's using %{MONTHNUM}, which will not match Feb. See github.com/logstash-plugins/logstash-patterns-core/blob/master/… for the full pattern. I think you'll have to create a custom pattern. — baudsp

Sufiyan Ghori Sufiyan Ghori · Accepted Answer · 2018-05-08T04:13:41

As @baudsp had suggested, you need to create a custom pattern for BIND9 log. For this you first need to know what every field actually mean,

The query log entry first reports a client object identifier in @0x format. Next, it reports the client's IP address and port number, and the query name, class and type. Next, it reports whether the Recursion Desired flag was set (+ if set, - if not set), if the query was signed (S), EDNS was in used along with the EDNS version number (E(#)), if TCP was used (T), if DO(DNSSEC Ok) was set (D), if CD (Checking Disabled) was set (C), if a valid DNS Server COOKIE was received (V), or if a DNS COOKIE option without a valid Server COOKIE was present (K). After this the destination address the query was sent to is reported. Note: This reflects BIND 9.11.0 behaviour.

so for your BIND9 query log,

17-Feb-2018 23:06:56.326 queries: info: client @0x563d72c3ea20 172.26.0.1#34564 (test.example.com): query: test.example.com IN A +E(0)K (172.26.0.3)

the pattern would be,

%{MONTHDAY:day}[-]%{MONTH}[-]%{YEAR}\s*%{TIME}\s*%{WORD:queries}[:]\s*%{WORD:info}[:]\s*%{WORD:client}\s*%{DATA:client_data}\s*%{IP:client_ip}[#]%{NUMBER:client_port}\s*\(%{HOSTNAME}\)[:]\s*query:\s*%{HOSTNAME:query_value}\s*%{WORD}\s*%{WORD:record_type}\s*%{NOTSPACE:misc}\s*\(%{IP:destination}\)

This will generate following output,

{
  "day": [
    [
      "27"
    ]
  ],
  "MONTH": [
    [
      "Feb"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "TIME": [
    [
      "23:06:56.326"
    ]
  ],
  "HOUR": [
    [
      "23"
    ]
  ],
  "MINUTE": [
    [
      "06"
    ]
  ],
  "SECOND": [
    [
      "56.326"
    ]
  ],
  "queries": [
    [
      "queries"
    ]
  ],
  "info": [
    [
      "info"
    ]
  ],
  "client": [
    [
      "client"
    ]
  ],
  "client_data": [
    [
      "@0x563d72c3ea20"
    ]
  ],
  "client_ip": [
    [
      "172.26.0.1"
    ]
  ],
  "IPV6": [
    [
      null,
      null
    ]
  ],
  "IPV4": [
    [
      "172.26.0.1",
      "172.26.0.3"
    ]
  ],
  "client_port": [
    [
      "34564"
    ]
  ],
  "BASE10NUM": [
    [
      "34564"
    ]
  ],
  "HOSTNAME": [
    [
      "test.example.com"
    ]
  ],
  "query_value": [
    [
      "test.example.com"
    ]
  ],
  "WORD": [
    [
      "IN"
    ]
  ],
  "record_type": [
    [
      "A"
    ]
  ],
  "misc": [
    [
      "+E(0)K"
    ]
  ],
  "destination": [
    [
      "172.26.0.3"
    ]
  ]
}

Grok pattern for BIND9 log

1 Answers