Container killed by YARN for exceeding memory limits in Spark Scala

0
votes

I am Facing below Error while Running my Spark Scala code using Spark-submit command.

ERROR cluster.YarnClusterScheduler: Lost executor 14 on XXXX: Container killed by YARN for exceeding memory limits. 55.6 GB of 55 GB physical memory used.

The Code of the Line Number it throws the error is below...

df.write.mode("overwrite").parquet("file")

I am Writing a Parquet file.... It was working till yesterday not sure from last run only it is throwing the error with same input file.

Thanks, Naveen

1
scalaapache-spark
Are you data from Hive or HDFS ? It seems, the issue is not in your program but the no of executors your program is grabbing is less than what is required by your application. Your admin might have set limit for no of memory per executor which is exceeding. Also is spark dynamic allocation is turned on or off ? - tarun
Writing it to HDFS ... Regarding Spark dynamic allocation i am using below conf in my spark-submit command --conf spark.dynamicAllocation.enabled= - Naveen Kumar
can you check your dag ,at what stage of transformation is executor getting killed ? Are you using and shuffling operation which is causing this ? - tarun
Acually i did use --conf spark.dynamicAllocation.enabled=false is this fine? or should i need to use true? - Naveen Kumar
You application requires more executors make --conf spark.dynamicAllocation.enabled=true. You job should work. - tarun

1 Answers

0
votes

By Running with below conf in spark-submit command, the issues is resolved and code ran successfully.

--conf spark.dynamicAllocation.enabled=true

Thanks, Naveen