spark-shell ERROR YarnScheduler: Lost executor

Question

Inside of spark-shell I keep getting the following error message in between my comment submitting. What are these possibly about?

scala> 16/04/12 12:49:14 ERROR YarnScheduler: Lost executor 2 on XXXXXXXX: remote Rpc client disassociated 16/04/12 12:49:14 ERROR YarnScheduler: Lost executor 5 on XXXXXXXX: remote Rpc client disassociated 16/04/12 12:49:16 ERROR YarnScheduler: Lost executor 4 on XXXXXXXX: remote Rpc client disassociated

This is too broad to answer but first thing I'd look into is the memory overhead properties given to yarn by Spark — eliasah
I agree, and I will definitely check the memory overhead properties given by yarn log. — Ruxi Zhang

J Maurer J Maurer · Accepted Answer · 2016-04-16T16:24:33

If you have dynamic resource allocation configured, these errors will show up when unneeded resources are released to Yarn. ERROR level seems misleading to me, perhaps INFO would be more appropriate. http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation

edit: This should be fixed in Spark 1.6 https://issues.apache.org/jira/browse/SPARK-4134

spark-shell ERROR YarnScheduler: Lost executor

1 Answers