How many partitions does Spark create when a file is loaded from S3 bucket?

Question

If the file is loaded from HDFS by default spark creates one partition per block. But how does spark decide partitions when a file is loaded from S3 bucket?

Answer 1

See the code of org.apache.hadoop.mapred.FileInputFormat.getSplits().

Block size depends on S3 file system implementation (see FileStatus.getBlockSize()). E.g. S3AFileStatus just set it equals to 0 (and then FileInputFormat.computeSplitSize() comes into play).

Also, you don't get splits if your InputFormat is not splittable :)

Answer 2

By default spark will create partitions of size 64MB when reading from s3. So a 100 MB file will be split into 2 partitions, 64MB and 36MB. An object having size less than or equal to 64 MB wont be split at all.

Answer 3

Even when reading a file from an S3 bucket, Spark (by default) creates one partition per block i.e. total no of partitions = total-file-size / block-size.

The value of block size for S3 is available as a property in Hadoop's core-site.xml file which is used by Spark:

<property>
  <name>fs.s3a.block.size</name>
  <value>32M</value>
  <description>Block size to use when reading files using s3a: file system.
  </description>
</property>

Unlike HDFS, AWS S3 is not a file system. It is an object store. The S3A connector makes S3 look like a file system.

Please checkout the documentation for more details.