Getting started - Spark, IPython notebook with pyspark

0
votes

I want to understand how to set up everything to be able to have a iPython notebook and pyspark working.

I currently have a ec2 instance.

Tell me if I'm wrong but I think I can either use this :

./spark-ec2 --key-pair=awskey --identity-file=awskey.pem --region=us-west-1 --zone=us-west-1a --spark-version=1.1.0 launch my-spark-cluster

(spark.apache.org)

or this :

aws emr create-cluster --name "Spark cluster" --release-label emr-4.1.0 --applications Name=Spark --ec2-attributes KeyName=myKey --instance-type m3.xlarge --instance-count 3 --use-default-roles

(http://docs.aws.amazon.com/)

Is there really a big difference between those two ?

To write and execute my scripts, I should be on the master node, right ?

So, the ec2 instance I'm using to get Spark running won't be used ?

(I have to install the ipython notebook on the master node ?)

Thanks in advance for your help.

(I've looked for tutorials but I did not found exactly what I was looking for.

Do you know good ones ?)

1
pythonhadoopapache-sparkipython-notebookpyspark

1 Answers

0
votes

The first tutorial that you mentioned is the best that I have found to date. I'm not sure about the difference between the two methods that you mentioned.

I used ./spark-ec2 to successfully build a spark cluster on EC2. Getting it to work with IPython required a few extra installations.

First launch your cluster with the ./spark-ec2 process. I had success with the following:

  1. Install Anaconda and PY4J on master and all worker nodes. Anaconda should be set at the front of the path (eg.PATH=/home/ec2-user/anaconda2/bin:$PATH).

  2. Next, ssh to master, then $cd /root/spark to run:

    MASTER=spark://<public DNS>:7077 PYSPARK_PYTHON=~/anaconda2/bin/python PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS='notebook  --ip="*"' ./bin/pyspark
  3. Visit < master public DNS >:8888 to gain access to the notebook. If you want this to be public, then open up port 8888 in the security group for master. Otherwise, create an ssh tunnel.