Hazelcast Predicate gets stuck during Heavy Load

Question

I have a two node Hazelcast cluster with 6 gb heap size each . I have a predicate which runs on four fields so for example purpose , lets consider a class Employee

  public class Employee {
  String id,
  String name,
  String surname,
  String timestamp
  .....
  }

The class has in total like 13 fields or so . I am running a range query on timestamp and absolute matching with the other 3 fields - id , name and surname. For serialization , I am using IdentifiedDataSerializable as that is the most efficient form of serialization hazelcast has to offer. I have a tomcat servlet container setup , so every request coming in fires a predicate on the cluster . The issue I am currently facing is when there are like around 100,000 records in the cluster and I conduct a performance test on the tomcat container, most of the tomcat threads get stuck as the predicate query never returns . I have looked at the threading model provided by hazelcast - https://docs.hazelcast.org/docs/latest-dev/manual/html-single/index.html#threading-model . I have tinkered with the different kind of threads using the properties in the documentation and it has improved things but it has been basically firing in the dark . I have added an index on the field id but that too isn't really improving things .

I would really appreciate it , if someone points me in the right direction on how I could go about solving this issue. Thank you in advance!

EDIT -

Hazelcast version used both for the cluster and client is 3.9 . Also , i am using hazelcast as embedded in a spring boot application . Don't think that will have some effect , but wanted to let you all know.

Can you check the thread state of servers in thread dumps? Where are they stuck? Hazelcast servers also print Diagnostic logs that are also helpful in investigating stuck situations, for they will tell you what operations are stuck and piling on. Also check servers for their GC logs, it could well be gc pauses that is blocking server JVMs from any processing. Adding indexes on the fields that you are using in predicate helps boost the performance by a great deal. — wildnez
A word of caution - I wouldn’t start messing with internal executors until we know whats causing the predicate to block and where, and every other solution has failed. — wildnez
Also, it would be helpful if you share the predicate code and map configuration. — Ozan Kılıç
Unfortunately i wont be able to share the predicate code , All i can tell you is that I am using a predicateBuilder and EntryObject to create the predicate . And as i have explained in the question , it is absolute matching on 3 fields and range query on timestamp. — Indraneel Bende

Gokhan Oner Gokhan Oner · Accepted Answer · 2019-02-12T18:12:34

@Indraneel-Bende, a couple of suggestions:

Hazelcast query engine evaluates all predicates & combine the results. If it's an AND predicate, after all 4 Predicate results retrieved, common ones on all 4 results will be selected. So, if any of those fields have low cardinality and return too many results, this will slow down the predicate. So, I'd suggest defining an index on only 1 field that has the highest cardinality, or at most 2 fields.
Since not all fields will be indexed, then entries returned from the index store need to be deserialized to apply non-indexed predicates. Even with IdentifiedDataSerializable, if you have too many fields, you'll pay full deserialization cost every time. Instead, you can implement Portable serialization. Although the stored entry size will be bigger, This way Hazelcast members can deserialize just the fields that you're using in those predicates and it'll speed up the queries.
As described here, https://docs.hazelcast.org/docs/3.9/manual/html-single/index.html#copying-indexes, index results are copied to make sure results are correct, especially new nodes join/leave the cluster. If you're sure that membership won't change, you can set hazelcast.index.copy.behavior to NEVER. This will also speed up the queries.

I personally suggest testing them one by one. My first suggestion will definitely improve query speed. If you need further performance, try the second approach & see the difference. Since the last one can cause wrong or duplicate results if your membership changes while querying, I'd use it with caution.

Hazelcast Predicate gets stuck during Heavy Load

1 Answers