Fluentd Elasticsearch Kibana logs dynamic index GEOPOINT mapping

Question

I am using EFK logging stack and have managed to set up fluentd filter and match configurations so that the tomcat access logs collected in Kibana all needed fields are there: IP, latitude, longitude, City, country code, etc.

I have curl -XPUT -H "Content-Type: application/json" --data @fluentd_mapping.json http://$host/fluentd/fluentd/_mapping but it only affects the fluentd index. Not the dynamically created daily-log index <foo>.access.logs.*

So my question is how to apply the mapping so that the geo_point type is effective for the daily log index? Thanks.

I restarted the ES container, curl -XPUT the following mapping.json which is picked up by fluentd index but not the daily-log index:

{
    "fluentd": {
        "location_array": {
          "type": "geo_point"
        },
        "location_properties": {
          "type": "geo_point"
        },
        "location_string": {
          "type": "geo_point"
        },
    }
}

Existing templates:

                "dynamic_templates": [{
    "kibana_index_template:.kibana": {
                "dynamic_templates": [{
    "security-index-template": {
                "dynamic_templates": [{
                        "path_match": "result\\.(input(\\..+)*|(transform(\\..+)*)|(actions\\.transform(\\..+)*))\\.search\\.request\\.(body|template)",
    "logstash-index-template": {

Existing index patterns:

        "index_patterns": [".monitoring-logstash-6-*"],
        "index_patterns": [".monitoring-kibana-6-*"],
        "index_patterns": [".ml-anomalies-*"],
        "index_patterns": [".kibana"],
        "index_patterns": [".ml-state"],
        "index_patterns": [".watches*"],
        "index_patterns": [".monitoring-beats-6-*"],
        "index_patterns": [".monitoring-es-6-*"],
        "index_patterns": [".security_audit_log*"],
        "index_patterns": [".ml-meta"],
        "index_patterns": [".security-*"],
        "index_patterns": [".triggered_watches*"],
        "index_patterns": [".watcher-history-9*"],
        "index_patterns": [".monitoring-alerts-6"],
        "index_patterns": [".ml-notifications"],
        "index_patterns": [".logstash"],

I added the following template but to no avail:

        "fluentd": {
                "order": 0,
                "index_patterns": ["myapp.access.logs*"],
                "settings": {},
                "mappings": {
                        "fluentd": {
                                "properties": {
                                        "location_array": {
                                                "type": "geo_point"
                                        },
                                        "location_properties": {
                                                "type": "geo_point"
                                        },
                                        "location_string": {
                                                "type": "geo_point"
                                        }
                                }
                        }
                },
                "aliases": {}
        },

It's not picked up by myapp.access.logs-

In the template you added, just change "fluentd" to "_doc" and it should work. — Opster Elasticsearch Expert

Opster Elasticsearch Expert Opster Elasticsearch Expert · Accepted Answer · 2019-02-11T06:51:05

Please specify the content of fluentd_mapping.json for clarity, but it looks like the indices have different name pattern and that is the reason the mapping is not applied to all the indices.

you should use elasticsearch template in order to configure the geoip mapping. in the template take a look at "index_patterns" which describe the indices matching condition. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

once the index_patterns regex will match all the indices you need (including the access logs index) then the GEOIP mapping and all the rest will get applied as expected.

Fluentd Elasticsearch Kibana logs dynamic index GEOPOINT mapping

1 Answers