0
votes

I am working on a project that requires to run multiple jobs simultaneously (on different data).

I have a graph with many connected components that have huge sizes. I'm creating new job for every connected component and submitting it. Will these jobs run simultaneously or they will be queued up, and run in a FIFO priority?

Of course I can create only one job and send each connected component to each mapper/reducer but this will cause the program to run slowly, as I am doing heavy computation on each connected component (inversion etc.)

1

1 Answers

0
votes

Jobs will run simultaneously based on the resource availability in the cluster unless there are no dependent jobs.

When more jobs were submitted which involves complex calculations only the resource in the cluster determines whether the program runs slow or fast.