I have deployed a HDInsight 3.5 Spark (2.0) cluster on Microsoft Azure with the standard configurations (Location = US East, Head Nodes = D12 v2 (x2), Worker Nodes = D4 v2 (x4)). Locally I have installed sparkmagic following the steps in https://github.com/jupyter-incubator/sparkmagic/blob/master/README.md#installation and https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-jupyter-notebook-install-locally and changed the config.json file. When starting jupyter notebook I can chose the PySpark kernel. Even tough I get the message that the kernel is ready, when I try to execute a simple statement (e.g. t = 4), the kernel starts to run infinitely. Could you provide possible solution(s)?
1 Answers
1
votes
Most probably, this is an issue where the config.json is configured with the wrong endpoint, username, or password. If you are using the base64 password field, make sure the password is base64 encoded.
Without more information regarding errors (log file should be in ~/.sparkmagic/logs), it's hard to say why you couldn't connect.
