Spark read csv file submitted from --files

Question

I'm submitting a Spark job to a remote spark cluster on yarn and including a file in the spark-submit --file I want to read the submitted file as a dataframe. But I'm confused about how to go about this without having to put the file in HDFS:

spark-submit \
--class com.Employee \
--master yarn \
--files /User/employee.csv \
--jars SomeJar.jar

spark: SparkSession = // create the Spark Session
val df = spark.read.csv("/User/employee.csv")

Answer 1

spark.sparkContext.addFile("file:///your local file path ")

Add file using addFile so that it can be available at your worker nodes. Since you want to read local file in cluster mode.

You may need to do a slight change according to scala and the spark version your are using.

Answer 2

employee.csv is in the work directory of executor, just reading it as follows:

val df = spark.read.csv("employee.csv")