Notebooks on EMR (AWS): Failed to start kernel

Question

I am creating a notebook with the built-in option from "Amazon EMR" following the steps in the official documentation

• Everything as default
• EMR-Cluster created with the notebook-Wizzard

After starting successfully the notebook in Jupyter, an error appears by loading the kernel. Different kernels (Python3, PySpark) produce the same result. By clicking in "Kernel Error" on the toolbar in Jupyter, following message is shown:

Failed to start kernel

"Error attempting to connect to Gateway server url 'http://localhost:17777'. Ensure gateway url is valid and the Gateway instance is running."

Any clue what am I missing?

Thanks!

Answer 1

I encountered the same issue a while back. Check your emr release version when you launch the cluster. There seems to be an issue with 5.30.0. I was able to connect using 5.29.0. Otherwise, you can host your own notebook.

Answer 2

I am getting this error when signing in as root user.

I signed in as IAM user (see here https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html) and everything works fine.

Answer 3

For aws emr 6, same issue persists. Kernel error in 6.2.0, but using 6.1.0 would work.Also agree with the answer above in IAM user, I switched from root user to IAM then 6.2.0 worked fine.

So basically you either choose to create and log in with IAM user for 6.2.0, or stay with root user but use 6.1.0.