How does mllib code run on spark?

1
votes

I am new to distributed computing, and I'm trying to run Kmeans on EC2 using Spark's mllib kmeans. As I was reading through the tutorial I found the following code snippet on http://spark.apache.org/docs/latest/mllib-clustering.html#k-means enter image description here

I am having trouble understanding how this code runs inside the cluster. Specifically, I'm having trouble understanding the following:

  1. After submitting the code to master node, how does spark know how to parallelize the job? Because there seem to be no part of the code that deals with this.

  2. Is the code copied to all nodes and executed on each node? Does the master node do computation?

  3. How do node communitate the partial result of each iteration? Is this dealt inside the kmeans.train code, or is the spark core takes care of it automatically?

1
apache-sparkapache-spark-mllib

1 Answers

1
votes
  1. Spark divides data to many partitions. For example, if you read a file from HDFS, then partitions should be equal to partitioning of data in HDFS. You can manually specify number of partitions by doing repartition(numberOfPartitions). Each partition can be processed on separate node, thread, etc. Sometimes data are partitioned by i.e. HashPartitioner, which looks on hash of the data.

Number of partitions and size of partitions generally tells you if data is distributed/parallelized correctly. Creating partitions of data is hidden in RDD.getPartitions methods.

Resource scheduling depends on cluster manager. We can post very long post about them ;) I think that in this question, the partitioning is the most important. If not, please inform me, I will edit answer.

  1. Spark serializes clusures, that are given as arguments to transformations and actions. Spark creates DAG, which is sent to all executors and executors execute this DAG on the data - it launches closures on each partition.

  2. Currently after each iteration, data is returned to the driver and then next job is scheduled. In Drizzle project, AMPLab/RISELab is creating possibility to create multiple jobs on one time, so data won't be sent to the driver. It will create DAG one time and schedules i.e. job with 10 iterations. Shuffle between them will be limited / will not exists at all. Currently DAG is created in each iteration and job in scheduled to executors

There is very helpful presentation about resource scheduling in Spark and Spark Drizzle.