Number of Partitions of Spark Dataframe

Question

Can anyone explain about the number of partitions that will be created for a Spark Dataframe.

I know that for a RDD, while creating it we can mention the number of partitions like below.

val RDD1 = sc.textFile("path" , 6) 

But for Spark dataframe while creating looks like we do not have option to specify number of partitions like for RDD.

Only possibility i think is, after creating dataframe we can use repartition API.

df.repartition(4)

So can anyone please let me know if we can specify the number of partitions while creating a dataframe.

Answer 1

You cannot, or at least not in a general case but it is not that different compared to RDD. For example textFile example code you've provides sets only a limit on the minimum number of partitions.

In general:

• Datasets generated locally using methods like range or toDF on local collection will use spark.default.parallelism.
• Datasets created from RDD inherit number of partitions from its parent.

• Datsets created using data source API:

  • In Spark 1.x typically depends on the Hadoop configuration (min / max split size).
  • In Spark 2.x there is a Spark SQL specific configuration in use.
• Some data sources may provide additional options which give more control over partitioning. For example JDBC source allows you to set partitioning column, values range and desired number of partitions.

Answer 2

Default number of shuffle partitions in spark dataframe(200)

Default number of partitions in rdd(10)