java priority queue to queue adaptation

Question

I have a priority queue that lists aload of jobs from an sql database in date order. I then have get the closestDeadlineJob function below that gets the top job, checks if any other jobs have the same date then compare priorities to see which is the top job. I then get returned the top job.

Find original queue top job:

public JobRequest closestDeadlineJob(int freeCPUS) {
        // find top job to determine if other jobs for date need to be considered
        JobRequest nextJob = scheduledJobs.peek(); // return top most job

        if (nextJob != null) {

            System.out.println("Found top EDF job:");
            printJob( nextJob );

            // what is it's date?
            Date highestRankedDate = nextJob.getConvertedDeadlineDate();

            // create a temporary queue to work out priorities of jobs with same deadline
            JobPriorityQueue schedulerPriorityQueue = new JobPriorityQueue();

            // add the top job to priority queue
            //schedulerPriorityQueue.addJob(nextJob);

            for (JobRequest jr : scheduledJobs) {

                // go through scheduled jobs looking for all jobs with same date
                if (jr.getConvertedDeadlineDate().equals(highestRankedDate)) {
                    // same date deadline, soadd to scheduler priority queue
                    schedulerPriorityQueue.addJob(jr);
                    System.out.println("Adding following job to priority queue:");
                    printJob(jr);
                }
            }

            JobRequest highestPriorityJob = schedulerPriorityQueue.poll();
            // this is the item at the top of the PRIORTY JOB queue to return 

            // remove that item from scheduledJobs
            scheduledJobs.remove(highestPriorityJob);


            return highestPriorityJob;
        } else {
            return null;
        }
    }

following code to process top jobs into a queue:

    public void processNextJob() {
        /*
         * 1. get # of free CPU's still avaialble
         * 2. get top most job from priority queue
         * 3. run job - put to CPU queue
         * 4. develop a CPU queue here
         * 5. count cores against freeCPUS and some sort of calculation to sort run times
         */
        int freeCPUS = 500;
        int availableCPUS = 0;
        Queue q = new PriorityQueue();

//        while(freeCPUS >= 500)
//        {
//           
//        }


        JobRequest nextJob = schedulerPriorityQueue.closestDeadlineJob(freeCPUS); // returns top job from queue
        if (nextJob != null) {
            System.out.println("Top priority / edf job:");
            System.out.print(nextJob.getUserID() + "-->");
            System.out.print(nextJob.getStartDate() + "--START-->");
            System.out.print(nextJob.getEndDate() + "---END-->");
            System.out.print(nextJob.getDeadDate() + "--DROP-->");
            System.out.print(nextJob.getDepartment() + "-->");
            System.out.print(nextJob.getProjectName() + "-->");
            System.out.print(nextJob.getProjectApplication() + "-->");
            System.out.print(nextJob.getPriority() + "--PRIORITY-->");
            System.out.print(nextJob.getCores() + "-->");
            System.out.print(nextJob.getDiskSpace() + "-->");
            System.out.println(nextJob.getAnaylsis());

            // now got correct job based on earliest deadline / priority
            // implement a FIFO queue here / execution stack
            // add next job here
        } else {
            System.out.println("Job = null");
        } 

    }

What I need to do is fix my poor attempt or adaptation at putting jobs from my closestDeadlineJob into a queue then stop putting them in a queue when I reach my 500 core limit. at the moment I just get stuck in the for loop below the while true and I don't think the way I've set out would even work after leaving the loop.

Any thoughts?

EDIT

public void processNextJob() {
        /*
         * 1. get # of free CPU's still avaialble
         * 2. get top most job from priority queue
         * 3. run job - put to CPU queue
         * 4. develop a CPU queue here
         * 5. count cores against freeCPUS and some sort of calculation to sort run times
         */
        int freeCPUS = 500;
        int availableCPUS = 0;

        JobRequest nextJob = schedulerPriorityQueue.closestDeadlineJob(freeCPUS); // returns top job from queue
        if (nextJob != null) {
            System.out.println("Top priority / edf job:");
            printJob( nextJob );
            // go through scheduled jobs looking for all jobs with same date
            if (nextJob.getCores() <= freeCPUS) {
                // same date deadline, soadd to scheduler priority queue
                schedulerPriorityQueue.addJob(nextJob);
                System.out.println("Adding following job to execution queue:");
                printJob( nextJob );     
                // can use this to get the next top job but need to add calculations to printout the next top job aslong as CPU less than 500
//                schedulerPriorityQueue.closestDeadlineJob(freeCPUS);
//                schedulerPriorityQueue.addJob(nextJob);
            } else if (nextJob.getCores() > freeCPUS) {
                System.out.println("Queue temporarily full");
            }
            // now got correct job based on earliest deadline / priority
            // implement a FIFO queue here / execution stack
            // add next job here
        } else {
            System.out.println("Job = null");
        }

    }

I imagine I need to implement a loop above and move out the if statements saying take next job, if under 500, loop through again and get another then put it into a new queue of some sort, when 500 cores criteria is met stop adding to the new queue

Not really answering the question, but is there a reason you can't do more filtering of the data in SQL? — SimonC
Also, what should be the behaviour when all 500 (!?) cores are busy? Should new jobs be rejected, or queued up? — SimonC
Due to restraints and how things are set up no, I just need to add "x" amount of jobs to a queue(don't know which type is best) thats core limit is under 500 and when it can't add anymore from the queue or it's full then stop adding until space becomes free. they will stay in the original queue until space becomes available to put them in the "execution queue" which I'm trying to set up here — Curious_Bop
so the way its set up is I have a priorityQueue thats sorted and each item in the priortyQueue has it's own cores, some have 100 or some have slightly more or less, I just want to add "x" amount of jobs to a queue from my closestDeadlineJob where that sorts out the next top job, and when the .getCores reaches 500 or near enough so the next job can't run then stop adding from the priortyqueue to the queue — Curious_Bop
You should do all this sorting and grouping directly in the SQL and cut out all the malarkey. — user207421

SimonC SimonC · Accepted Answer · 2013-02-21T14:38:53

I would use the utilities in the java.util.concurrent package as much as possible.

To start, ou can define a PriorityBlockingQueue with a Comparator that sorts you jobs by date then priority, so the job with the earliest date and highest priority is always at the start of the queue:

PriorityBlockingQueue<JobRequest> q = new PriorityBlockingQueue<Test1.JobRequest>(0, new Comparator<JobRequest>()
  {
    @Override
    public int compare(JobRequest o1, JobRequest o2)
    {
      int dateComparison = o1.getDate().compareTo(o2.getDate());
      if (dateComparison != 0)
        return dateComparison;
      // assume higher number means higher priority
      return o2.getPriority() - o1.getPriority();
    }
  });

I'm still not sure I understand your requirements on the cores, but you have two options here. If you want up to 500 jobs to execute concurrently, then reject new items you can use an executor with SynchronousQueue:

ExecutorService executor = new ThreadPoolExecutor(0 /*core size*/, 
                                                  500 /*max size*/, 
                                                  0 /*keep alive*/, 
                                                  TimeUnit.SECONDS, 
                                                  new SynchronousQueue<Runnable>());

Alternatively, if you want fewer jobs executing concurrently, you could use an ArrayBlockingQueue which blocks while it is full:

ExecutorService executor = new ThreadPoolExecutor(0 /*core size*/, 
                                                  5 /*max size*/, 
                                                  0 /*keep alive*/, 
                                                  TimeUnit.SECONDS, 
                                                  new ArrayBlockingQueue(500-5)<Runnable>());

Then pull jobs from the queue and execute them, handling rejected execution however you want to:

while (!isFinished)
{
  JobRequest job = q.take();
  try
  {
    executor.execute(job);
  }
  catch (RejectedExecutionException e)
  {

  }
}

If, however, you just want 500 jobs running concurrently and subsequent jobs queued, just pass in a LinkedBlockingQueue or, use one of the utility methods on Executors, like newFixedThreadPool(int nThreads).

java priority queue to queue adaptation

2 Answers