Python vs Scala (for Spark jobs)

Question

I am pretty new to Spark, currently exploring it by playing with pyspark and spark-shell.

So here is the situation, I run same spark jobs with pyspark and spark-shell.

This is from pyspark:

textfile = sc.textFile('/var/log_samples/mini_log_2')
textfile.count()

And this one from spark-shell:

textfile = sc.textFile("file:///var/log_samples/mini_log_2")
textfile.count()

I tried both of them several times, first (python) one takes 30-35 seconds to complete while second one (scala) takes about 15 seconds. I am curious about what may cause this different performance results? Is it because of choice of language or spark-shell do something in background that pyspark don't?

UPDATE

So I did some tests on larger datasets, about 550 GB (zipped) in total. I am using Spark Standalone as master.

I observed that while using pyspark, tasks are equally shared among executors. However when using spark-shell, tasks are not shared equally. More powerful machines get more tasks while weaker machines gets fewer tasks.

With spark-shell, job is finished in 25 minutes and with pyspark it is around 55 minutes. How can I make Spark Standalone assign tasks with pyspark, as it assigns tasks with spark-shell?

Answer 1

Using python has some overhead, but it's significance depends on what you're doing. Though recent reports indicate the overhead isn't very large (specifically for the new DataFrame API)

some of the overhead you encounter relates to constant per job overhead - which is almost irrelevant for large jobs. You should to do a sample benchmark with a larger data set, and see if the overhead is a constant addition, or if it's proportional to the data size.

Another potential bottleneck is operations that apply a python function for each element (map, etc.) - if these operations are relevant for you, you should test them too.