Managing changes to model objects in hazelcast?

Question

The general architecture of the system is a lightweight app server that connects to hazelcast and a common model dependency between the two. The app can go down but the business requires that the hazelcast cluster must remain up between upgrades (to not lose contextual session data).

If I have a Map defined in hazelcast:

        <hz:map name="Fruit"
                backup-count="${hazelcast.default.backup-count}"
                time-to-live-seconds="${hazelcast.default.time-to-live-seconds}"
                max-idle-seconds="${hazelcast.default.max-idle-seconds}"
                eviction-policy="${hazelcast.default.eviction-policy}"
                max-size="${hazelcast.default.max-size}"
                eviction-percentage="${hazelcast.default.eviction-percentage}"
                in-memory-format="${hazelcast.in-memory-format}"/>

Which stores a class called Fruit:

public class Fruit implements Serializable {

    private static final long serialVersionUID = -7184021704842980361;

    //some fields and methods
}

Let's say v1 of the app, model and hazelcast server are all deployed and are working beautifully in production. Then new requirements come out of the woodwork and I'm forced to make changes to the Fruit class (updating serialVersionUID to denote changes to the class):

private static final long serialVersionUID = -5284413340344918080L;

public boolean isPoisonous(){
    return determineToxicityOnHumans();
}

The problem is that when I deploy a new version of the api that depends on v2 of the model (where the Fruit class exists), I get an exception:

com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.InvalidClassException: 

io.anew.hz.model.Fruit; local class incompatible: stream classdesc serialVersionUID = -5284413340344918080, local class serialVersionUID = -7184021704842980361

This makes sense to me as the serialVersionUID has changed from version 1 to version 2. The problem I envision and am currently struggling to wrap my head around is managing this change without causing downtime for the cluster.

My question is this: what are some other folks' experiences with regards to managing model changes to hazelcast's object dependencies? What are some working strategies for managing divergent model changes to the cluster? Is migration a valid strategy?

Answer 1

You might want to have a look at custom serializers where you can deal with versioning. So by adding a version to the stream when serializing, you know how to deal with it when deserializing.

Look for (Identified)DataSerializable, StreamSerializer and Portable. https://github.com/hazelcast/hazelcast-code-samples/tree/master/serialization

Answer 2

If you are adding new fields to a class, this cannot, for obvious reasons, be done using Java serialization - a serialized object of one version cannot be deserialized to another without potential data loss (even if serialVersionUID is fudged).

Unfortunately I don't think Hazelcast's Portable serialization mechanism, plays nicely if you try to read a field that doesn't exist in the stream (which will happen when you start rolling your new class out through the cluster), nor does it like you reading the remaining bytes of the object stream, so you're going to have to roll-you-own Custom serialization mechanism.

The important concept you need to support (to borrow from Coherence) is one of future data.

Future data refers to the fact that when de-serializing an object to an old version of the class, there won't be fields to hold all the values. If the object doesn't have a way to hold these values then they will potentially be lost should this object be pushed back to the cache (or moved from one node to another).

What your custom serialization needs to support is the ability to read known values from an array of bytes plus the remaining unknown values as a byte array. This byte array containing the unknown values can then be stored in the object and written back when it is re-serialized.

Once this mechanism is working, you just need to do a rolling upgrade, where each cache node is upgraded in turn.