Hazelcast FREE_HEAP_PERCENTAGE policy not working as expected

Question

Hazelcast's FREE_HEAP_PERCENTAGE max size policy is not evicting map entries when the minimum configured free heap percentage on the JVM is met.

Below is a test case that illustrates this issue. If the test is run with other policy types (including USED_HEAP_PERCENTAGE), it works as expected such that entries are evicted when this max size is reached. However, if it is run with the FREE_HEAP_PERCENTAGE, entries are never evicted from the map, despite the fact that the free heap percentage is beyond the percentage allowed (as can be confirmed if running the management center, for example).

public class HazelcastMaxSizeTest {

    private static final String GROUP_NAME = "TEST";
    private static final String MAP_NAME = "test";

    private static MaxSizeConfig POLICY_THAT_WORKS = new MaxSizeConfig(5, MaxSizeConfig.MaxSizePolicy.USED_HEAP_PERCENTAGE);
    private static MaxSizeConfig POLICY_THAT_DOES_NOT_WORK = new MaxSizeConfig(99, MaxSizeConfig.MaxSizePolicy.FREE_HEAP_PERCENTAGE);

    public static void main(String[] args) throws Exception {

        HazelcastInstance instance = startHazelcast("hazelcast1", POLICY_THAT_DOES_NOT_WORK);
        System.out.println("started " + instance.getName());

        IMap<Long, byte[]> map = createMap(instance, MAP_NAME);
        System.out.println("map size: " + map.size());

        instance.shutdown();
    }

    private static HazelcastInstance startHazelcast(String instanceName, MaxSizeConfig maxSizeConfig) {

        MapConfig mapConfig = new MapConfig(MAP_NAME);
        mapConfig.setMaxSizeConfig(maxSizeConfig);
        mapConfig.setStatisticsEnabled(true);
        mapConfig.setEvictionPolicy(EvictionPolicy.LRU);
        mapConfig.setMinEvictionCheckMillis(0L);
        mapConfig.setBackupCount(1);

        Config config = new Config(instanceName);
        config.addMapConfig(mapConfig);
        config.getManagementCenterConfig().setEnabled(true);
        config.getManagementCenterConfig().setUrl("http://localhost:8080/mancenter");
        config.getGroupConfig().setName(GROUP_NAME).setPassword(GROUP_NAME);
        return Hazelcast.getOrCreateHazelcastInstance(config);
    }

    private static IMap<Long, byte[]> createMap(HazelcastInstance instance, String mapname) {

        IMap<Long, byte[]> map = instance.getMap(mapname);

        map.addEntryListener(new EntryEvictedListener<Long, byte[]>() {

            @Override
            public void entryEvicted(EntryEvent<Long, byte[]> event) {
                System.out.println("evicted " + event.getName() + ": " + event.getKey());
            }

        }, false);

        for (long i = 1; i <= 50_000; i++) {
            map.set(i, new byte[100]);
            if (i % 1000 == 0) {
                System.out.println("set " + map.getName() + ": " + i);
            }
        }

        return map;
    }

}

After some further debugging, I am wondering if Hazelcast's EvictionChecker may be incorrectly calculating free heap percentage. While debugging, I found that the calculated free heap percentage was greater than 100. It's calculation uses available / total, but as described here, this should perhaps be available / max instead?

Is there something wrong with my configuration or expected use of the FREE_HEAP_PERCENTAGE policy? Or is this a Hazelcast bug?

Specs: Hazelcast 3.6, -Xmx256m

what was your heap size? I used your test, set -Xmx256m, and it worked for me (can see evicted messages). BTW for testing purposes setting minEvictionCheckMillis to zero makes much sense, by setting it zero, eviction is checked whether it is doable upon every map#set. — mrck
Thanks. I've edited my question to use 0 for the minEvictionCheckMillis and specify -Xmx256m which does not work for me. I was finally able to get the FREE_HEAP_PERCENTAGE policy to evict some entries after significantly increasing the size of the map, however. What I don't understand, though, is that (1) if the USED_HEAP_PERCENTAGE is consistently met at 2% (and up to at least 5% or more), why is the FREE_HEAP_PERCENTAGE not reached at 99% (or even 100%)? (2) the management center showed ~30% heap usage percentage, so why would a policy requiring 99% free heap not be met? — shelley
After some further debugging, I am suspicious that Hazelcast may not be calculating the free heap correctly. I've updated my question accordingly. — shelley
When i checked the picture here, i can see that the calculated amount of free heap can be found with max - (total - free) and it is the equivalent of max - total + free, so EvictionChecker seems correct on this. — mrck
can you please elaborate on that, why are those in-equivalent? — mrck

shelley shelley · Accepted Answer · 2016-02-24T20:45:56

After further debugging, I found the unexpected behavior of the FREE_HEAP_PERCENTAGE policy was due to a bug in Hazelcast's calculation of the available memory. This issue has been reported as Issue 7599 in Hazelcast's issue tracker.

Hazelcast FREE_HEAP_PERCENTAGE policy not working as expected

1 Answers