I'm using elasticsearch on my local machine. The data directory is only 37MB in size but when I check logs, I can see:
[2015-05-17 21:31:12,905][WARN ][cluster.routing.allocation.decider] [Chrome] high disk watermark [10%] exceeded on [h9P4UqnCR5SrXxwZKpQ2LQ][Chrome] free: 5.7gb[6.1%], shards will be relocated away from this node
Quite confused about what might be going wrong. Any help?
From Index Shard Allocation:
... watermark.high controls the high watermark. It defaults to 90%, meaning ES will attempt to relocate shards to another node if the node disk usage rises above 90%.
The size of your actual index doesn't matter; it's the free space left on the device which matters.
If the defaults are not appropriate for you, you've to change them.
To resolve the issue in which, the log is recorded as:
high disk watermark [90%] exceeded on [ytI5oTyYSsCVfrB6CWFL1g][ytI5oTy][/var/lib/elasticsearch/nodes/0] free: 552.2mb[4.3%], shards will be relocated away from this node
You can update the threshold limit by executing following curl request:
curl -XPUT "http://localhost:9200/_cluster/settings" \
-H 'Content-Type: application/json' -d'
{
"persistent": {
"cluster": {
"routing": {
"allocation.disk.threshold_enabled": false
}
}
}
}'
this slightly modified curl command from the Elasticsearch 6.4 docs worked for me:
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "2gb",
"cluster.routing.allocation.disk.watermark.high": "1gb",
"cluster.routing.allocation.disk.watermark.flood_stage": "500mb",
"cluster.info.update.interval": "1m"
}
}
'
if the curl -XPUT command succeeds, you should see logs like this in the Elasticsearch terminal window:
[2018-08-24T07:16:05,584][INFO ][o.e.c.s.ClusterSettings ] [bhjM1bz] updating [cluster.routing.allocation.disk.watermark.low] from [85%] to [2gb]
[2018-08-24T07:16:05,585][INFO ][o.e.c.s.ClusterSettings ] [bhjM1bz] updating [cluster.routing.allocation.disk.watermark.high] from [90%] to [1gb]
[2018-08-24T07:16:05,585][INFO ][o.e.c.s.ClusterSettings ] [bhjM1bz] updating [cluster.routing.allocation.disk.watermark.flood_stage] from [95%] to [500mb]
[2018-08-24T07:16:05,585][INFO ][o.e.c.s.ClusterSettings ] [bhjM1bz] updating [cluster.info.update.interval] from [30s] to [1m]
https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html
Its a warning and won't affect anything. The storage processors (SPs) use high and low watermarks to determine when to flush their write caches. The possible solution can be to free some memory
And the warning will disappear. Even with it showing, the replicas will not be assigned to the node which is okay. The elasticsearch will work fine.
Instead of percentage I use absolute values and rise values for better space use (in pre-prod):
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.disk.threshold_enabled": true,
"cluster.routing.allocation.disk.watermark.low": "1g",
"cluster.routing.allocation.disk.watermark.high": "500m",
"cluster.info.update.interval": "5m"
}
}
Also I reduce pooling interval to make ES logs shorter ))
