Hazelcast - Inconsistent throughput for distributed map

1
votes

I am using Hazelcast version 3.3 (open source) and facing an issue with inconsistent throughput for distributed map.

I have a single cache instance and at one end an application is putting data in cache (at a rate of 1000 objects every 1 min) and at other end another application is getting data from the map (at a rate of few 100 every sec). Java objects are using default Java Serialization and in-memory-format as BINARY.

Fetching of data works fine for majority of “get” operations where it takes less than 1 msec, however sometime “get” operation takes up-to 20 msecs to fetch the same data. This extreme behavior is not acceptable as I can’t delay processing because a random fetch took longer than expected.

I am looking for forward for predictable/consistent throughput in fetching data; a random spike in fetching data from Hazelcast instance is not acceptable. If someone has came across the same problem and has pointers please share.

2
hazelcast

2 Answers

1
votes

do you have GC log? It might very well be caused by a pause introduced by garbage collector. On HotSpot (Sun/Oracle Java) can enable the log by starting JVM with these flags:

-Xloggc:/path/to/file/gc.log -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCDetails

It will produce a log with GC events, so you can try to correlate the unusually high latencies with GC pauses.

If you can also try to use this swich:

-XX:+PrintGCApplicationStoppedTime

It will report all JVM pauses, not just the pauses triggered by Garbage Collector.

1
votes

I have made 2 change on the project.

1st I have changed serialization from java.io.Serializable to com.hazelcast.nio.serialization.DataSerializable 2nd I have introduced java warm up time. I am accessing complete map few times at start up time before actaully using in applicaiotn.

With these changes I can see good improvement in the performance.