Spark Streaming and Data Locality when dynamically loading files

1
votes


I'm running a spark streaming app the receives file paths on HDFS from Kafka and should open these files and perform some kind of computation on them. The problem is that I cannot enjoy the benefits of data locality as the executors might run on any node and the executor that opens the file is not necessarily the one who holds it.

Is there a way to dynamically open files in the manner I presented while maintaining data locality ?

Thanks,
Daniel

2
hadoopapache-sparkspark-streaming

2 Answers

0
votes

I'm not sure what you mean by opening the file, and it would be helpful if you can share some code, but if you are using sc.textFile that is an RDD transformation. Transformations are being scheduled as tasks by the cluster manager and therefore will not necessarily executed from the executor node that runs your DStream transformation.

0
votes

It seems that for the time being the suggested solution cannot support Data Locality.
To benefit from Data Locality we wrote a Kafka Consumer that initiates a normal spark job with file paths retrieved from Kafka and the we use computePreferredLocations to ensure containers are allocated on the correct nodes:

val locData = InputFormatInfo.computePreferredLocations
(Seq(new InputFormatInfo(conf, classOf[TextInputFormat], new Path(“myfile.txt”)))
 val sc = new SparkContext(conf, locData)