Do I need to use Spark with YARN to achieve NODE LOCAL data locality with HDFS?

0
votes

Do I need to use Spark with YARN to achieve NODE LOCAL data locality with HDFS?
If I use Spark standalone cluster manager and have my data distributed in HDFS cluster, how will Spark know that data is located locally on the nodes?

1
hadoopapache-sparkhdfs

1 Answers

2
votes

YARN is a resource manager. It deals with memory and processes, and not with the workings of HDFS or data-locality.

Since Spark can read from HDFS sources, and the namenodes & datanodes take care of all that HDFS block data management outside of YARN, then I believe the answer is no, you don't need YARN. But you already have HDFS, which means you have Hadoop, so why not take advantage of integrating Spark into YARN?