In spark: What does repartition exactly work?

Question

In spark: after a groupBy everything goes to one executor. If a do a repartition(x) after it, does the rdd get distributed in x executors or is just chunked in the x number of partitions getting 1 executor with x rdd blocks?

Example:

rdd = rdd_tmp.groupBy.repartition(32).cache()
rdd.count()

If I'm using 32 executors and I run a groupBy operation, and cached the rdd. Do i get:

1. 1 executor with 32 rdd blocks

2. 32 executors with 1 rdd each

?

Answer 1

First run rdd_tmp.getNumPartitions(). If partition is less than 32, you can increase partition to 32 by using repartition(32). If you have 32 executors, each executor will work on 1 partition [there will be 32 tasks and each executor will be assigned 1 task].

In your case, 1 executor will get 1 block.

Note: If rdd_tmp.getNumPartitions() is greater than 32, repartition(32) will not work. Use coalesce(32) to decrease your partition.