How to configure Hazelcast MapStore chunk sizes?

Question

With the following configuration

<map name="event">
    <in-memory-format>OBJECT</in-memory-format>
    <backup-count>1</backup-count>
    <map-store enabled="true" initial-mode="EAGER">
        <class-name>xxx.EventMapStore</class-name>
        <write-delay-seconds>5</write-delay-seconds>
        <write-batch-size>5000</write-batch-size>
    </map-store>
</map>

and a straight forward MapStore implementation Hazelcast loads/stores entries from the persistent store in chunk sizes, which it seems to determine on it's own. On a single node for a key-size of around 20K (which are all loaded fine at once by loadAllKeys) it stores in chunks of around 200 and loads about 80 at a time (the size is different on every call). Which is a performance-nightmare.

Neither the write-batch-size nor the write-delay-seconds seem to have an effect for storage. Setting the advanced property hazelcast.map.load.chunk.size does not seem to have any effect on the loading chunk-size either.

How can a custom chunk-size for loading / storing be achieved?

And (just out of interest) how is this dynamic chunk-size being determined for every load/store?

Some logs from the loading (the first line comes from loadAllKeys and then the chunking, which Hazelcast does on its own starts ...):

2015-05-08 00:08:14 DEBUG EventMapStore:220 - Loading entire key-batch of 19876
2015-05-08 00:08:15 DEBUG EventMapStore:149 - Loading event-batch of 55
2015-05-08 00:08:17 DEBUG EventMapStore:149 - Loading event-batch of 78
2015-05-08 00:08:18 DEBUG EventMapStore:149 - Loading event-batch of 79
2015-05-08 00:08:19 DEBUG EventMapStore:149 - Loading event-batch of 72
2015-05-08 00:08:20 DEBUG EventMapStore:149 - Loading event-batch of ...

I guess it is possible to configure it to eg load in chunks of say 5000 and store with the same batch-size ... Right?

Apparently assigning more memory (a lot more) to the JVM cranks up the batch-size. It still doesn't meet the configuration, but it comes close. Anyway, the assigned memory size looks frightening when projected on the actual expectation in production ... — Simeon Isainmdom

ali ali · Accepted Answer · 2015-05-08T13:37:57

Hazelcast split the data into partitions and loading process handled by each partition. Since default partition-count is 271 each partition will have around 73 (20K/271) entries which is smaller than the batch-size. Use a smaller batch-size (smaller than 73) to see if it works as expected

How to configure Hazelcast MapStore chunk sizes?

1 Answers