spark schedulingMode

Question

I am building a Spark App, in which I submit several jobs (pyspark). I am using threads to run them in parallel, and also I am setting: conf.set("spark.scheduler.mode", "FAIR")

Still, I see the jobs run serially in FIFO way. Am I missing something?

EDIT: After writing to the Spark Mailing List, I got a couple of things more:

• Spark's FairSchedulingAlgorithm is not round robin: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulingAlgorithm.scala#L43
• There is no preemption, that means even using scheduler.mode 'FAIR' you cannot be sure each stage will be executed 'a bit'. The scheduling happens just when resources (core) are available.

I was totally missing this last point

Answer 1

Apparently, spark doesn't support threading for all kinds of jobs. You can only submit a job in parallel if it is a spark action.

Inside a given Spark application (SparkContext instance), multiple parallel jobs can run simultaneously if they were submitted from separate threads. By “job”, in this section, we mean a Spark action (e.g. save, collect) and any tasks that need to run to evaluate that action. Spark’s scheduler is fully thread-safe and supports this use case to enable applications that serve multiple requests (e.g. queries for multiple users).

please follow this link

Answer 2

How many executors do you have?

If you have 1 executor then FIFO is same as FAIR.

I am saying so because by default standalone mode creates 2 executors and in "cluster" mode you job would take 1 for driver and 1 for executor.

So you need 4 executors to run 2 jobs in cluster mode.