heap memory usage in elasticsearch

Question

Currently I'm working on a project which uses elasticsearch 1.4.1.

The cluster consists of 22 nodes, and 20 of them are data nodes, each node has 16GB as the heap memory.

The thing is, when I'm doing massive queries, some of the nodes(2 or 3) consume 70% of the heap memory, while the rest just use less than 10%.

So I'm wondering is this because most of the queries go to those 2 or 3 nodes?

If not, can I further achieve better performance?

Thanks!

Just updated:

I just ran this command: curl -XGET localhost:9200/_cat/shards?v, and it returned:

index              shard prirep state       docs    store ip         node
....
mm                 2     r      STARTED  2248969  293.6mb 10.2.4.117 Mark Todd           
mm                 2     p      STARTED  2248969  293.6mb 10.2.4.129 Saint Elmo          
mm                 19    r      STARTED 30172116    3.5gb 10.2.4.126 Fixer               
mm                 19    p      STARTED 30172116    3.5gb 10.2.4.123 Loki  
....

I'm wondering what the store here means? if it is the actual size of the documents, can I load all of them into memory?

Can you also give a bit more information on the amount of shards? What mechanism do you use for routing over the shards, the default? — Jettro Coenradie
@JettroCoenradie Currently I have 28 shards, 20 data nodes. The routing is based on the first few characters of a street name, which will definitely result in uneven distribution But for now it seems there is not any better solution. — milodky
The street name is the search condition, so at least it will lead to the correct shard. — milodky

Vineeth Mohan Vineeth Mohan · Accepted Answer · 2015-01-01T02:02:15

This could be because the document matches for that query is somehow colonized to just those 2 machines. That is , if there are 20 million matches , chances are that 8 million match belongs to one machine , 8 million match comes from another and only just 2 million comes from rest of the 18 machines. I am guessing you are also using aggregation in the process due to which field data cache is fabricated.

heap memory usage in elasticsearch

1 Answers