Settle the right number of partition on RDD

Question

I read some comments which says than a good number of partition for a RDD is 2-3 time the number of core. I have 8 nodes each with two 12-cores processor, so i have 192 cores, i setup the partition beetween 384-576 but it doesn't seems works efficiently, i tried 8 partition, same result. Maybe i have to setup other parameters in order to my job works better on the cluster rather than on my machine. I add that the file i analyse make 150k lines.

 val data = sc.textFile("/img.csv",384)

"doesn't seems works efficiently" is a pretty generic description. Could you provide some details? — zero323
It works more slowly on 8 nodes rather than on my single machine — KyBe
Well, 150k is not much (what is the size of the dataset?) Unless you do really complex transformations coordination and communication overhead may be simply larger than a cost of performing task locally. — zero323
I have an other file with 2m lines and 5 columns, how can i set the partition number accordingly ? — KyBe
Did yo try the 2-3*cores for your large file? But to be honest, 2 million isn't large either. What computation are you doing on each line? — The Archetypal Paul

Aakash Aggarwal Aakash Aggarwal · Accepted Answer · 2017-09-29T07:17:37

The primary effect would be by specifying too few partitions or far too many partitions.

Too few partitions You will not utilize all of the cores available in the cluster.
Too many partitions There will be excessive overhead in managing many small tasks.

Between the two the first one is far more impactful on performance. Scheduling too many smalls tasks is a relatively small impact at this point for partition counts below 1000. If you have on the order of tens of thousands of partitions then spark gets very slow.

Now, considering your case, you are getting the same results from 8 and 384-576 partitions. Generally the thumb rule says, NoOfPartitions = (NumberOfWorkerNodes*NoOfCoresPerWorkerNode)-1 It says that, as we know, the task is processed by CPU cores. So we should set that many number of partitions which is the total number of cores in the cluster to process-1(for Application Master of driver). That means the each core will process each partition at a time. That means with 191 partitions can improve the performance. Otherwise impact of setting less and more partitions scenario is explained in beginnning.

Hope this will help!!!

Settle the right number of partition on RDD

1 Answers