Ignite Remote Cluster [Failed to map keys for cache (all partition nodes left the grid)]

Question

I am having some issues creating cache K,V pairs on my remote Ignite cluster. My Ignite nodes are built on a Maven project in Java8, clustered via multicast configuration, with partitioned cache. The problem faced here is that ignite can only successfully create and retrieve cache via SqlFieldQuery or NoSQL when there are exactly 1 node in the cluster. When I scale it to more than 1(in this case 2), I face this error thrown by the remote server.

Caused by: org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException: Failed to map keys for cache (all partition nodes left the grid) [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=3], cache=Transactions]
at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.serverNotFoundError(GridPartitionedSingleGetFuture.java:711) ~[ignite-core-2.6.0.jar:2.6.0]
at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:332) ~[ignite-core-2.6.0.jar:2.6.0]
at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:216) ~[ignite-core-2.6.0.jar:2.6.0]
at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.init(GridPartitionedSingleGetFuture.java:208) ~[ignite-core-2.6.0.jar:2.6.0]
at org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.getAsync(GridDhtColocatedCache.java:246) ~[ignite-core-2.6.0.jar:2.6.0]
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4556) ~[ignite-core-2.6.0.jar:2.6.0]
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4537) ~[ignite-core-2.6.0.jar:2.6.0]
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1350) ~[ignite-core-2.6.0.jar:2.6.0]
at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:907) ~[ignite-core-2.6.0.jar:2.6.0]

This is my server setup:

public class App 
{
    protected static IgniteConfiguration config;
    protected static String igniteID = UUID.randomUUID().toString();

    public static void main( String[] args )
    {
        System.out.println( "Starting instance of Ignite..." );
        config = new IgniteConfiguration();
        config.setIgniteInstanceName("ignite-node-"+igniteID);
        config.setClientMode(false);
        setupDiscoveryConfig();
        setupDataConfig();
        //cacheConfig();
        config.setPeerClassLoadingEnabled(false);
        config.setDeploymentMode(DeploymentMode.CONTINUOUS);
        config.setPeerClassLoadingMissedResourcesCacheSize(0);
        config.setFailureDetectionTimeout(10000);
        config.setClientFailureDetectionTimeout(10000);
        config.setNetworkTimeout(10000);
        Ignite ignite = Ignition.start(config);
        loadAndRemove(ignite);

    }

    private static void loadAndRemove(Ignite ignite) {
        ignite.cluster().active();
        IgniteCache<String, TransactionRecord> cache = ignite.getOrCreateCache("Transactions");
    }

    private static void cacheConfig(){
        CacheConfiguration cacheConfig = new CacheConfiguration("Transactions");
        cacheConfig.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
        cacheConfig.setCacheMode(CacheMode.PARTITIONED);
        config.setCacheConfiguration(cacheConfig);
    }


    private static void setupDataConfig(){
        DataStorageConfiguration storageConfiguration = new DataStorageConfiguration();
        storageConfiguration.getDefaultDataRegionConfiguration()
                .setPersistenceEnabled(true);
        config.setDataStorageConfiguration(storageConfiguration);
    }

    private static void setupDiscoveryConfig(){
        TcpDiscoverySpi spi = new TcpDiscoverySpi();
        TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
        ((TcpDiscoveryMulticastIpFinder) ipFinder).setMulticastGroup("228.10.10.157");
        ipFinder.setAddresses(Arrays.asList("127.0.0.1"));
        spi.setIpFinder(ipFinder);
        config.setDiscoverySpi(spi);
    }
}

Client configs as shown below:

@Bean
    public Ignite igniteInstance(){
        IgniteConfiguration config = new IgniteConfiguration();
        config.setIgniteInstanceName("cluster-client");
        config.setClientMode(true);
        config.setPeerClassLoadingEnabled(false);
        config.setDeploymentMode(DeploymentMode.CONTINUOUS);
        config.setPeerClassLoadingMissedResourcesCacheSize(0);
        config.setFailureDetectionTimeout(10000);
        config.setClientFailureDetectionTimeout(10000);
        config.setNetworkTimeout(10000);
        TcpDiscoverySpi spi = new TcpDiscoverySpi();
        TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
        ipFinder.setMulticastGroup("228.10.10.157");
        ipFinder.setAddresses(Arrays.asList("127.0.0.1"));
        spi.setIpFinder(ipFinder);
        config.setDiscoverySpi(spi);
        return Ignition.getOrStart(config);
    }

Data insertion is using the simple cache.put command. Appreciate if anyone can help me out on this.

Answer 1

Most probably, your cluster has only one baseline node, which means, that all data is stored on this node only. When it leaves the cluster, all cache operations start throwing this exception, because there are no affinity nodes left.

To avoid it, you should activate the cluster only when all server nodes are started. If topology changes, you should either bring back the failed nodes, or reset to baseline topology to trigger partition reassignment and rebalancing.

Here is how to set the baseline topology from code:

Collection<ClusterNode> nodes = ignite.cluster().forServers().nodes();
ignite.cluster().setBaselineTopology(nodes);

Documentation on baseline topology feature: https://apacheignite.readme.io/docs/baseline-topology

Answer 2

Deactivating the Cluster

Deactivation deallocates all memory resources, including your application data, on all cluster nodes and disables public cluster API. If you have in-memory caches that are not backed up by a persistent storage, you will lose the data and will have to repopulate these caches. To deactivate the cluster, use one of the following methods:

// Connect to the cluster.
Ignite ignite = Ignition.start();

// Deactivate the cluster. 
ignite.cluster().active(false);

Setting the Topology From Code

As explained above, baseline topology is automatically initialized when you activate the cluster manually. Use the IgniteCluster.activate() method to activate the cluster from code. Then, you can use the IgniteCluster.setBaseLineTopogy() method to adjust an existing baseline topology. Note that the cluster must be activated for the method to be called.

Setting Baseline Topology Nodes Setting Cluster Topology as Baseline Topology

 // Connect to the cluster.
Ignite ignite = Ignition.start();

// Activate the cluster.
// This is required only if the cluster is still inactive.
ignite.cluster().active(true);

// Get all server nodes that are already up and running.
Collection<ClusterNode> nodes = ignite.cluster().forServers().nodes();

// Set the baseline topology that is represented by these nodes.
ignite.cluster().setBaselineTopology(nodes);

If you update the baseline topology later, let's say by adding new nodes to it, then Ignite will rebalance the data automatically across all the baseline nodes.