Spark: How to specify the number of executors to hold an RDD?

Question

I am trying to maximize parallelism by distributing the RDD to as many executors as possible. As far as I know, the user can use repartition, coalesce or parallelize to change the number of partitions. But I am not able to find a way to change the number of executors to hold the partitions. Can any one hint on how it could be done?

Answer 1

When you start your spark app. There is a parameter --num-executors to specifying how many executors you want, and in parallel, --executor-cores is to specify how many tasks can be executed in parallel in each executors.

In your case, you can specify a big number of executors with each one only has 1 executor-core. Then for example you have 10 partitions and 10 executors, then each executors will be assigned one task to process one partition.

Answer 2

Directly type spark-submit in command line, you will get manual. It contains something below

 Spark standalone with cluster deploy mode only:
  --driver-cores NUM          Cores for driver (Default: 1).

 Spark standalone or Mesos with cluster deploy mode only:
  --supervise                 If given, restarts the driver on failure.
  --kill SUBMISSION_ID        If given, kills the driver specified.
  --status SUBMISSION_ID      If given, requests the status of the driver specified.

 Spark standalone and Mesos only:
  --total-executor-cores NUM  Total cores for all executors.

 Spark standalone and YARN only:
  --executor-cores NUM        Number of cores per executor. (Default: 1 in YARN mode,
                              or all available cores on the worker in standalone mode)

 YARN-only:
  --driver-cores NUM          Number of cores used by the driver, only in cluster mode
                              (Default: 1).
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --num-executors NUM         Number of executors to launch (Default: 2).
                              If dynamic allocation is enabled, the initial number of
                              executors will be at least NUM.
  --archives ARCHIVES         Comma separated list of archives to be extracted into the
                              working directory of each executor.
  --principal PRINCIPAL       Principal to be used to login to KDC, while running on
                              secure HDFS.
  --keytab KEYTAB             The full path to the file that contains the keytab for the
                              principal specified above. This keytab will be copied to
                              the node running the Application Master via the Secure
                              Distributed Cache, for renewing the login tickets and the
                              delegation tokens periodically.

It depends on your deployment mode. You have to submit your script to spark with specific parameters above to define number of executors accordingly.