Partitioning in spark while reading from RDBMS via JDBC

14
votes

I am running spark in cluster mode and reading data from RDBMS via JDBC.

As per Spark docs, these partitioning parameters describe how to partition the table when reading in parallel from multiple workers:

  • partitionColumn
  • lowerBound
  • upperBound
  • numPartitions

These are optional parameters.

What would happen if I don't specify these:

  • Only 1 worker read the whole data?
  • If it still reads parallelly, how does it partition data?
1
apache-sparkjdbcapache-spark-sqlpartitioning
Do you have sample code to read records from RDBMS via jdbc? – Surender Raja
@SurenderRaja you can check - gist.github.com/devender-yadav/5c4328918602b7910ba883e18b68fd87 – Dev

1 Answers

26
votes

If you don't specify either {partitionColumn, lowerBound, upperBound, numPartitions} or {predicates} Spark will use a single executor and create a single non-empty partition. All data will be processed using a single transaction and reads will be neither distributed nor parallelized.

See also: