Hazelcast eviction is not working with FREE_HEAP_PERCENTAGE policy

Question

I'm using hazelcast 3.12.2 version. Below is the code snippet that I used;

package com;

import com.hazelcast.config.Config;
import com.hazelcast.config.EvictionPolicy;
import com.hazelcast.config.MapConfig;
import com.hazelcast.config.MaxSizeConfig;
import com.hazelcast.core.EntryEvent;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IMap;
import com.hazelcast.map.listener.EntryEvictedListener;

public class HazelcastMaxSizeTest {

    private static final String GROUP_NAME = "TEST";

    private static final String MAP_NAME = "test";

    private static final String MAP_NAME1 = "test1";

    private static final String MAP_NAME2 = "test2";

    private static MaxSizeConfig POLICY_THAT_DOES_NOT_WORK = new MaxSizeConfig(
            20,
            MaxSizeConfig.MaxSizePolicy.FREE_HEAP_PERCENTAGE);

    public static void main(String[] args) throws Exception {

        HazelcastInstance instance = startHazelcast("hazelcast1", POLICY_THAT_DOES_NOT_WORK);
        System.out.println("started " + instance.getName());

        IMap<Long, byte[]> map = createMap(instance, MAP_NAME, 10000000);
        System.out.println("map size: " + map.size());

        IMap<Long, byte[]> map1 = createMap(instance, MAP_NAME1, 15000000);
        System.out.println("map1 size: " + map1.size());

        IMap<Long, byte[]> map2 = createMap(instance, MAP_NAME2, 20000000);
        System.out.println("map2 size: " + map2.size());

        instance.shutdown();
    }

    private static IMap<Long, byte[]> createMap(HazelcastInstance instance, String mapname, int bytes) {

        IMap<Long, byte[]> map = instance.getMap(mapname);

        map.addEntryListener(new EntryEvictedListener<Long, byte[]>() {

            @Override
            public void entryEvicted(EntryEvent<Long, byte[]> event) {
                System.out.println("evicted " + event.getName() + ": " + event.getKey());
            }

        }, false);

        for (long i = 1; i <= 10; i++) {
            map.set(i, new byte[bytes]);
        }

        return map;
    }

    private static HazelcastInstance startHazelcast(String instanceName, MaxSizeConfig maxSizeConfig) {

        MapConfig mapConfig = new MapConfig(MAP_NAME);
        mapConfig.setMaxSizeConfig(maxSizeConfig);
        mapConfig.setStatisticsEnabled(false);
        mapConfig.setEvictionPolicy(EvictionPolicy.LRU);
        mapConfig.setMinEvictionCheckMillis(0L);
        mapConfig.setBackupCount(1);

        MapConfig mapConfig1 = new MapConfig(MAP_NAME1);
        mapConfig.setMaxSizeConfig(maxSizeConfig);
        mapConfig.setStatisticsEnabled(false);
        mapConfig.setEvictionPolicy(EvictionPolicy.LRU);
        mapConfig.setMinEvictionCheckMillis(0L);
        mapConfig.setBackupCount(1);

        MapConfig mapConfig2 = new MapConfig(MAP_NAME2);
        mapConfig.setMaxSizeConfig(maxSizeConfig);
        mapConfig.setStatisticsEnabled(false);
        mapConfig.setEvictionPolicy(EvictionPolicy.LRU);
        mapConfig.setMinEvictionCheckMillis(0L);
        mapConfig.setBackupCount(1);

        Config config = new Config(instanceName);
        config.addMapConfig(mapConfig);
        config.addMapConfig(mapConfig1);
        config.addMapConfig(mapConfig2);
        config.getGroupConfig().setName(GROUP_NAME).setPassword(GROUP_NAME);
        return Hazelcast.getOrCreateHazelcastInstance(config);
    }

}

When I try to run the above code with 450mb heap space it is ending up with 'Out of Memory' error. After debugging I knew that eviction is not working even if the heap space used is 80%. Please let me know if I'm configured incorrectly. Any suggestion will be genuinely helpful.

Neil Stevenson Neil Stevenson · Accepted Answer · 2020-03-16T17:58:51

This code has a cut & paste error

        MapConfig mapConfig1 = new MapConfig(MAP_NAME1);
        mapConfig.setMaxSizeConfig(maxSizeConfig);
        mapConfig.setStatisticsEnabled(false);
        mapConfig.setEvictionPolicy(EvictionPolicy.LRU);
        mapConfig.setMinEvictionCheckMillis(0L);
        mapConfig.setBackupCount(1);

Should be

        MapConfig mapConfig1 = new MapConfig(MAP_NAME1);
        mapConfig1.setMaxSizeConfig(maxSizeConfig);
        mapConfig1.setStatisticsEnabled(false);
        mapConfig1.setEvictionPolicy(EvictionPolicy.LRU);
        mapConfig1.setMinEvictionCheckMillis(0L);
        mapConfig1.setBackupCount(1);

Same for mapConfig2.

Although that won't stop the first of the three maps from needing eviction.

Update 1: I can make eviction work. You need to set this before creating your Hazelcast instance to accommodate the small number of keys :

        config.setProperty("hazelcast.partition.count", "1");

Then try this code, if you set factor to 2 it works and eviction happens, set it to 1 and it doesn't.

        int factor = 2;
        byte[] value = new byte[bytes / factor];
        for (long i = 1; i <= 10 * factor; i++) {
            System.out.println("Write " + i + " size " + value.length + " to " + mapname + " free memory " +
                   (Runtime.getRuntime().freeMemory() / (1024 * 1024)) + "MB");
            map.set(i, value);
        }

However, when factor is set to 1 and -Xmx512m -Xms512m, I get an OOME when 139MB is free. This is more than 20% of the heap free, so eviction isn't applicable.

Update 2 I have logged this issue https://github.com/hazelcast/hazelcast/issues/16760 You can get an OOME with plenty memory left. This is what happened here, eviction hadn't been trigged as free memory was above the 20% configured.

Update 3 It's a G1 issue, see the Github issue, out of humongous space. You can try tuning, change GC or just as easily create 10 times as many objects that are 1/10th of the size and the problem goes away.

Hazelcast eviction is not working with FREE_HEAP_PERCENTAGE policy

1 Answers