Is it possible to see the Spark UI through EMR (through VPC)?

0
votes

I've gone through the process described in https://docs.aws.amazon.com/ElasticMapReduce/latest/ManagementGuide/emr-ssh-tunnel-local.html to open an ssh tunnel through local-port forwarding, and then going to http://localhost:8157/.

This brings up the Hadoop/YARN interface, at which point I click on the Tracking UI/ApplicationMaster link, and receive a "This site's server DNS could not be found" page.

According to the AWS docs http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-spark-history.html that should bring me to the Spark UI.

Is there another step I'm missing to be able to see the Spark UI?

1
amazon-web-serviceshadoopapache-spark

1 Answers

1
votes

The best experience comes from using a SOCKS proxy through SSH so that it can dynamically support the URLs and ports. Take a look at https://docs.aws.amazon.com/ElasticMapReduce/latest/ManagementGuide/emr-ssh-tunnel.html.