pyspark on yarn got permission denied with pipe

Question

i am trying to run a pyspark on yarn with CDH , in this pyspark main program, there is a statement like rdd.pipe("XXX.sh"),every time I run it, an error of permission denied will pop up, what should I do to resolve this error? thanks.

erro log info is as blow:

File "/data/yarn/nm/usercache/work/appcache/application_1495632173402_0079/container_1495632173402_0079_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func File "/data/yarn/nm/usercache/work/appcache/application_1495632173402_0079/container_1495632173402_0079_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func File "/data/yarn/nm/usercache/work/appcache/application_1495632173402_0079/container_1495632173402_0079_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func File "/data/yarn/nm/usercache/work/appcache/application_1495632173402_0079/container_1495632173402_0079_01_000001/pyspark.zip/pyspark/rdd.py", line 317, in func File "/data/yarn/nm/usercache/work/appcache/application_1495632173402_0079/container_1495632173402_0079_01_000001/pyspark.zip/pyspark/rdd.py", line 715, in func File "/usr/lib64/python2.6/subprocess.py", line 642, in init errread, errwrite) File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child raise child_exception OSError: [Errno 13] Permission denied

at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38

Torres Xie Torres Xie · Accepted Answer · 2017-07-18T06:56:49

finally I solved it by doing as follows chmod 777 -R /bashPath

The problem is Permission Denied.So First I think maybe the master or slave haven't the permission to execute the bash.But after I execute the command "chmod +x XXX.sh" and submit the task,The same error is there.Then I think if it is also need readable permissions.So I tried this.And this does work

pyspark on yarn got permission denied with pipe

1 Answers