Spark Job Internals

0
votes

I tried looking through the various posts but did not get an answer. Lets say my spark job has 1000 input partitions but I only have 8 executor cores. The job has 2 stages. Can someone help me understand exactly how spark processes this. If you can help answer the below questions, I'd really appreciate it

  1. As there are only 8 executor cores, will spark process the Stage 1 of my job 8 partitions at a time?
  2. If the above is true, after the first set of 8 partitions are processed where is this data stored when spark is running the second set of 8 partitions?
  3. If I dont have any wide transformations, will this cause a spill to disk?
  4. For a spark job, what is the optimal file size. I mean spark better with processing 1 MB files and 1000 spark partitions or say a 10MB file with 100 spark partitions?

Sorry, if these questions are vague. This is not a real use case but as I am learning about spark I am trying to understand the internal details of how the different partitions get processed.

Thank You!

1
apache-sparkpysparkaws-glue

1 Answers

0
votes

Spark will run all jobs for the first stage before starting the second. This does not mean that it will start 8 partitions, wait for them all to complete, and then start another 8 partitions. Instead, this means that each time an executor finishes a partition, it will start another partition from the first stage until all partions from the first stage is started, then spark will wait until all stages in the first stage are complete before starting the second stage.

The data is stored in memory, or if not enough memory is available, spilled to disk on the executor memory. Whether a spill happens will depend on exactly how much memory is available, and how much intermediate data results.

The optimal file size is varies, and is best measured, but some key factors to consider:

  1. The total number of files limits total parallelism, so should be greater than the number of cores.
  2. The amount of memory used processing a partition should be less than the amount available to the executor. (~4GB for AWS glue)
  3. There is overhead per file read, so you don't want too many small files.

I would be inclined towards 10MB files or larger if you only have 8 cores.