Hazelcast Jet with an IMap source and OBJECT in-memory format

Question

I have items in a Hazelcast IMap in OBJECT format, and I'm using a Jet aggregation operation with that IMap as a pipeline source. I was hoping, because of the OBJECT format, to avoid any serialisation/deserialisation of the items in my IMap during processing, the same way native Hazelcast entry processing and queries work. However, I can see that my items are in fact being serialised and then deserialised before being passed to my aggregator.

Is it possible to avoid the serialisation/deserialisation step when using Jet in this way? If so, how?

Answer 1

Yes, local map reader will always serialize/deserialize the entries. The only way I can think of to work around this is to use a custom source which uses map.localKeySet() and then use mapUsingIMap to do a join on those keys. The source would look like below:

SourceBuilder.batch("localKeys", c -> c.jetInstance().getMap("map"))
 .fillBufferFn((map, buf) -> {
    for (Object key : map.localKeySet(predicate)) {
        buf.add(key);
    }
    buf.close();
 }).distributed(1).build());