spark cluster setup in Digital Ocean

0
votes

I am trying to setup a spark cluster in DigitalOcean and have created a master and two slave nodes there; I have been unable to connect to the master from the pyspark method setMaster() even though there are unused executors and lot of RAM still available.

The error I get is: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources.

My spark-env.sh file in master looks like this: 

export SPARK_MASTER_HOST='<MASTER IP ADDRESS>'
export JAVA_HOME='/usr/lib/jvm/java-8-oracle'
export SPARK_LOCAL_IP='<MASTER IP ADDRESS>'

The spark-env.sh file in slave looks like this: 

export SPARK_MASTER_HOST='<MASTER IP ADDRESS>'
export JAVA_HOME='/usr/lib/jvm/java-8-oracle'
export SPARK_LOCAL_IP='<SLAVE IP ADDRESS>'

I tried to use the Private Ip for the SPARK_MASTER_HOST as well as for the SPARK_LOCAL_IP but the error refuses to go away. What am I doing wrong?

1
pyspark

1 Answers

0
votes

Probably you already fixed it. But below are the steps I followed and worked for me.

  1. Change directory to Spark->Conf directory
  2. Create copy of the file spark-env.sh.template and give name just spark-env.sh
  3. Open that file (nano spark-env.sh or in any other editor)
  4. Set value of SPARK_PUBLIC_DNS (SPARK_PUBLIC_DNS=XX.XX.XX.XX). This IP will be your DO's droplet public IP address.
  5. Allow 4040 port into firewall (Since the UI running in 4040 port, so you need to enable it. ufw enable 4040/tcp).
  6. Change your directory to SPARK_HOME.
  7. bin/spark-shell (This will open spark shell in your terminal)
  8. If this is successful, you will notice the Web UI is listening in http://XX.XX.XX.XX:4040 port.
  9. Once you get the scala prompt, open browser in your system and try to access that URL (http://XX.XX.XX.XX:4040) and Bingo!! your job url will be visible.

I hope this will help you. Please ignore if you already found the solution by your own.

Happy coding. Cheers!!