Spark master copies the additional libraries to worker automatically?

Question

I built the Spark cluster by standalone cluster configuration.

1. I set the PYSPARK_PYTHON like below.

PYSPARK_PYTHON=/usr/bin/python2.7

2. I installed Python package(SciPy, numpy) on Spark master only. Then I wrote code like below.

    from pyspark import SparkContext
    from sklearn import datasets, svm

    def run(sc):
        iris = datasets.load_iris()
        digits = [ datasets.load_digits(), datasets.load_digits()]

        def learn(x):
            clf = svm.SVC(gamma=0.001, C=100.)
            clf.fit(x.data[:-1], x.target[:-1] )
            return clf.predict(x.data[-1])

        return sc.parallelize(digits).map(learn).collect()

    if __name__ == '__main__':
        print run(SparkContext() )

And I submit this code to spark master by using spark-submit.

spark-submit --master spark://192.168.18.31:7077 /master/scipy_example.py

I thought this code will not work because I didn't install sci-py on worker. But it works. Is it right? I confirmed this code ran on cluster too by Web-UI.

Answer 1

I found that spark-submit copies the python library from master to worker.

See the code below.

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L485

Also I heard it's very popular but not documented feature.