EMR: Is the Spark application available at bootstrap time?

1
votes

I am trying to start an EMR cluster with Spark using the CLI, where I specify Spark as an application. I also have some bootstrap scripts that configure things like IPython notebooks on top of Spark. However, when I try to refer to common Spark locations in my bootstrap scripts (/usr/bin/spark or /usr/lib/spark/bin) I get not found errors.

Can someone help me understand what the sequence of events in EMR clusters is -- are applications installed after bootstrapping?

1
amazon-web-servicesapache-sparkemr
having the same problem. is there any way to start my ipython notebook server with spark automatically, once Spark becomes available? - DanT
@DanT The weird thing is that this used to work at some point, but then it stopped working, and now we have to manually start up IPython notebook on the master. However, EMR > 4.10 has Zeppelin out of the box, and that should be sufficient for all Python notebook purposes - tchakravarty
Did you able to solve this issue? Thanks! - user3294904
@user3294904 Not really. I manually installed after bootstrapping finished. - tchakravarty

1 Answers

0
votes

So applications are installed during bootstrapping. You can't refer to common Spark locations before Spark is bootstrapped.