Anaconda (4.2) python 2.7.14: ensure that workers are registered and have sufficient resources

Question

Can anyone help to fix the following code please?

import pyspark
from pyspark import SparkContext, SparkConf
conf = SparkConf()
conf.setMaster('yarn-cluster')
conf.setAppName('test')
sc = SparkContext.getOrCreate()
r = sc.textFile("data.csv")
r.collect()

It errors out with the following exception:

WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

I am expecting the collection result will be printed out.

Thanks.

ChaosPredictor ChaosPredictor · Accepted Answer · 2018-10-29T22:02:16

As i understand it's an open issue, you can find more information here:

https://community.hortonworks.com/questions/37247/initial-job-has-not-accepted-any-resources.html

And here:

https://github.com/databricks/spark-knowledgebase/issues/9

Anaconda (4.2) python 2.7.14: ensure that workers are registered and have sufficient resources

1 Answers