I am using EFK logging stack and have managed to set up fluentd filter and match configurations so that the tomcat access logs collected in Kibana all needed fields are there: IP, latitude, longitude, City, country code, etc.
I have curl -XPUT -H "Content-Type: application/json" --data @fluentd_mapping.json http://$host/fluentd/fluentd/_mapping
but it only affects the fluentd index. Not the dynamically created daily-log index <foo>.access.logs.*
So my question is how to apply the mapping so that the geo_point type is effective for the daily log index? Thanks.
I restarted the ES container, curl -XPUT
the following mapping.json which is picked up by fluentd index but not the daily-log index:
{
"fluentd": {
"location_array": {
"type": "geo_point"
},
"location_properties": {
"type": "geo_point"
},
"location_string": {
"type": "geo_point"
},
}
}
Existing templates:
"dynamic_templates": [{
"kibana_index_template:.kibana": {
"dynamic_templates": [{
"security-index-template": {
"dynamic_templates": [{
"path_match": "result\\.(input(\\..+)*|(transform(\\..+)*)|(actions\\.transform(\\..+)*))\\.search\\.request\\.(body|template)",
"logstash-index-template": {
Existing index patterns:
"index_patterns": [".monitoring-logstash-6-*"],
"index_patterns": [".monitoring-kibana-6-*"],
"index_patterns": [".ml-anomalies-*"],
"index_patterns": [".kibana"],
"index_patterns": [".ml-state"],
"index_patterns": [".watches*"],
"index_patterns": [".monitoring-beats-6-*"],
"index_patterns": [".monitoring-es-6-*"],
"index_patterns": [".security_audit_log*"],
"index_patterns": [".ml-meta"],
"index_patterns": [".security-*"],
"index_patterns": [".triggered_watches*"],
"index_patterns": [".watcher-history-9*"],
"index_patterns": [".monitoring-alerts-6"],
"index_patterns": [".ml-notifications"],
"index_patterns": [".logstash"],
I added the following template but to no avail:
"fluentd": {
"order": 0,
"index_patterns": ["myapp.access.logs*"],
"settings": {},
"mappings": {
"fluentd": {
"properties": {
"location_array": {
"type": "geo_point"
},
"location_properties": {
"type": "geo_point"
},
"location_string": {
"type": "geo_point"
}
}
}
},
"aliases": {}
},
It's not picked up by myapp.access.logs-