Kibana settings gets cleared daily

0
votes

ELK 5.5.2 installed by manual

From kibana.yml:

Kibana uses an index in Elasticsearch to store saved searches, visualizations and dashboards. Kibana creates a new index if the index doesn't already exist.

For some reason index gets cleared daily, and all visualisations and dashboards disappear

there are records from elasticsearch log:

[2017-09-13T00:14:52,126][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nAnqErn] [logstash-2017.09.11/CLJQRSDOTG2GrNZaPi0I4Q] deleting index
[2017-09-13T00:14:52,131][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nAnqErn] [logstash-2017.09.12/45Z96SAOT_GmDt0lNVyfKw] deleting index
[2017-09-13T00:14:52,131][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nAnqErn] [.kibana/vf8BVEpkQBmgYxj0K4c6hQ] deleting index
[2017-09-13T00:14:59,346][INFO ][o.e.c.m.MetaDataCreateIndexService] [nAnqErn] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [_default_, index-pattern, server, visualization, search, timelion-sheet, config, dashboard, url]

probably there are some log rotation, but I've not found any info in the docs. how to save kibana settings?

2
elasticsearchkibanaelastic-stack

2 Answers

0
votes

Someone or something is deleting the .kibana index (third line), which contains your visualizations and dashboards. They are in the right place, but you need to stop that deletion — it shouldn't happen.

Is that always happening at the same time (around 00:14:52)? Then I'd assume it's some bad automation. Alternatively, if this instance is publicly accessible, it might be some (ransomware) bot killing your data.

Limiting access with a firewall, a reverse proxy, or X-Pack security would probably be a good idea here.

0
votes

one of my scripts cleaned all data older then 2 days including kibana index, we could close this question, probably it will help someone, main idea that kibana settings - in elasticsearch too

curl -XDELETE "http://localhost:9200/_all" -H 'Content-Type: 
application/json' -d'
{
    "query": {
        "range" : {
            "@timestamp" : {
                "lt" :  "now-2d"
            }
        }
    }
}'

was needed to specify logsatsh-* index