One task per node only in Apache Ignite

Question

I'm relatively new to Apache Ignite. I'm using Ignite compute to distribute tasks to nodes. My goal is a task dispatcher that produces tasks and submits these only to nodes that are "free". One node can only do one task at a time. If all nodes have a task running, the dispatcher shall wait for the next node to become available and then submit the next task.

I can implement this with a queue and async Callables, however I wonder if there is an Ignite onboard class that does something like this? Not sure the class ComputeTaskSplitAdapter is what I need to look at, I'm not fully understanding its purpose.

Any help appreciated.

Server nodes can join and leave the cluster while tasks are distributed. Tasks can take different amount of time on the nodes and as soon as a server finishes a task it shall get the next task.

Here's my node code:

    JobStealingCollisionSpi spi = new JobStealingCollisionSpi();
    spi.setActiveJobsThreshold(1);

    IgniteConfiguration cfg = new IgniteConfiguration();
    cfg.setCollisionSpi(spi);

    Ignition.start(cfg);

And this is my job distribution code (for testing):

    JobStealingCollisionSpi spi = new JobStealingCollisionSpi();
    spi.setActiveJobsThreshold(1);

    IgniteConfiguration cfg = new IgniteConfiguration();
    cfg.setCollisionSpi(spi);

    Ignition.setClientMode(true);
    Ignite ignite = Ignition.start(cfg);

    for (int i = 0; i < 10; i++)
    {
        ignite.compute().runAsync(new IgniteRunnable()
        {
            @Override
            public void run()
            {
                System.out.print("Sleeping...");
                try
                {
                    Thread.sleep(10000);
                } catch (InterruptedException e)
                {
                    e.printStackTrace();
                }
                System.out.println("Done.");
            }
        });
    }

Dmitriy Dmitriy · Accepted Answer · 2017-05-08T16:18:46

Yes, Apache Ignite has direct support for it. Please take a look at the One-at-a-Time section in the Job Scheduling documentation: https://apacheignite.readme.io/docs/job-scheduling#section-one-at-a-time

Note that every server has its own waiting queue and servers will move to the next job in their queue immediately after they are done with a previous job.

If you would like even more aggressive scheduling, then you can take a look at Job-Stealing scheduling here: https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/collision/jobstealing/JobStealingCollisionSpi.html

With Job Stealing enabled, servers will still jobs from the job-queues on other servers once their own queue becomes empty. Most of the parameters are configurable.

One task per node only in Apache Ignite

1 Answers