Apache Spark ALS algorithm

0
votes

I want to run a movie recommendation app based on ALS algorithm on Apache Spark using Python I’m using Spark2.2.0-Hadoop2.7 I have one master and 2 workers When I want to run the app using this command 

Spark-submit  —master Spark://192.168.190.132:7077 —total-executor-cores 8 —executor-memory 2g engine.py

I get errors it says the ratings.csv file doesn’t exist( I checked the addres everything is correct)

error picture below

https://i.stack.imgur.com/dgK2Q.jpg

But when I use this command Spark-submit app.pyit works but fails after a while

I’m not using HDFS I load dataset locally Do I need to copy datasets to all worker nodes?

1
pythonapache-sparkpysparkapache-spark-mllib
When you aren't using a single point of contact like hdfs, you need to have the files avaiable on all worker nodes. - cronoik
@cronoik so I should copy datasets to all worker nodes or use HDFS ? If using HDFS I just need to give the HDFS address? - Mansoor
I recommend using the HDFS as it offers you a wide range of possibilities. - cronoik
I copied datasets to all worker nodes exactly the same location but it doesn’t read the ratings and it doesn’t show any errors, I will use HDFS - Mansoor

1 Answers

0
votes

you need to upload dataset to HDFS if you want to work as spark standalone spark.using webui for all worker nodes.using hdfs -put to upload on HDFS.