How can I configure a custom field to be aggregatable in Kibana?

Question

I am new to running the ELK stack. I have Logstash configured to feed my webapp log into Elasticsearch. I am trying to set up a visualization in Kibana that will show the count of unique users, given by the user_email field, which is parsed out of certain log lines.

I am fairly sure that I want to use the Unique Count aggregation, but I can't seem to get Kibana to include user_email in the list of fields which I can aggregate.

Here is my Logstash configuration:

filter {
  if [type] == "wl-proxy-log" {
    grok {
      match => {
        "message" => [
          "(?<syslog_datetime>%{SYSLOGTIMESTAMP}\s+%{YEAR})\s+<%{INT:session_id}>\s+%{DATA:log_message}\s+license=%{WORD:license}\&user=(?<user_email>%{USERNAME}\@%{URIHOST})\&files=%{WORD:files}",
        ]
      }
      break_on_match => true
    }
    date {
      match => [ "syslog_datetime", "MMM dd HH:mm:ss yyyy", "MMM  d HH:mm:ss yyyy" ]
      target => "@timestamp"
      locale => "en_US"
      timezone => "America/Los_Angeles"
    }
    kv {
      source => "uri_params"
      field_split => "&?"
    }
  }
}

output {
  elasticsearch {
    ssl => false
    index => "wl-proxy"
    manage_template => false
  }
}

Here is the relevant mapping in Elasticsearch:

{
  "wl-proxy" : {
    "mappings" : {
      "wl-proxy-log" : {
        "user_email" : {
          "full_name" : "user_email",
          "mapping" : {
            "user_email" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    }
  }
}

Can anyone tell me what I am missing?

BTW, I am running CentOS with the following versions:

Elasticsearch Version: 6.0.0, Build: 8f0685b/2017-11-10T18:41:22.859Z, JVM: 1.8.0_151
Logstash v.6.0.0
Kibana v.6.0.0

Thanks!

user2040570 user2040570 · Accepted Answer · 2017-12-18T22:02:37

I figured it out. The configuration was correct, AFAICT. The issue was that I simply hadn't refreshed the list of fields in the index in the Kibana UI.

Management -> Index Patterns -> Refresh Field List (the refresh icon)

After doing that, the field began appearing in the list of aggregatable terms, and I was able to create the necessary visualizations.

How can I configure a custom field to be aggregatable in Kibana?

1 Answers