Elasticsearch running in docker on VM - Index Crashes every hours

Question

My setup:

elasticsearch logstash and metricbeat running as containers in docker-swarm mode. Metricbeat sents its logs to logstash and is written into the elasticsearch database.

One of each parts.

The Database persists on the host with the

--mount

parameter.

After some time (about 1h or earlyer). The index from metricbeat turns red.

and i got the log on elasticsearch:

[email protected] | [2017-09-01T08:26:08,287][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[metricbeat-2017.09.01][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[metricbeat-2017.09.01][0]] containing [125] requests]"})

[email protected] | [2017-09-01T08:26:08,287][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>125}

I adjusted with template that there should only be one shard without replicas, but it did not help.

I tried a configuration with three different nodes, but still the same. i even tried different combinations of shard / replica count but not effect there.

_cat/shards shows:

metricbeat-2017.09.01 0 p STARTED 2998 1.6mb 10.143.14.2 tPPhaKt

Are you sure that your ES is still alive and didn't die because your Docker container lacks memory resources? — Val
it is. i have some other indices running and they are still updated and not red. — Marcel Wolf
Can you update your question with the result you get from curl -XGET localhost:9200/_cat/shards?v — Val
@Val i restarted the setup. the storage that i use is a glusterfs over three vm. maybe the realy poor I/O from this is the Problem. Could this be possible? — Marcel Wolf
Yes, that might well be. Still interested to see the result I asked for. — Val

Marcel Wolf Marcel Wolf · Accepted Answer · 2017-09-02T06:11:48

I tried Not to Expose the Data Directory to the host and it works Fine. I think this was some I/O Problem with the Storage behind.

Elasticsearch running in docker on VM - Index Crashes every hours

1 Answers