Amazon EMR Application Master web UI?

Question

I have started running PIG jobs on Amazon EMR using Hadoop YARN (AMI 3.3.1) however as there is no longer a job tracker in Yarn, I can't seem to be able to find a web UI so that I can track the number of Mappers and Reducers for a MapReduce job, when I try to access the Application Master link provided in the resource manager UI page, I am told that the page doesn't exists (Picture provided below).

Does anyone know how I can access a UI through my web browser that will show me the current job status in terms of number of mappers, reducers and the % completed for each etc?

Thanks

Answer 1

Once you click the ApplicationMaster link from ResourceManager webpage, you'll be redirected to ApplicationMaster web ui; as EMR uses EC2 instances and each EC2 instance has 2 IP addresses associated with it, one used for private communication and another for public. EMR uses private ip addresses (private DNS) to setup hadoop hence, you'll be redirected to a url like this:

http://10.204.137.136:9046/proxy/application_1423027388806_0003/

which you could see is pointing to instance's private ip address and hence your browser cannot resolve the ip address, you just have to replace the private ip address with the public ip address (or public dns name) of that instance:

Obtaining the public ip address of an instance

1. Using the EC2 web interface

   You could login to the AWS EC2 console and find the instance's ip address's

   

2. Using the console:

   If you are logged into the instance and want to know it's public ip address then issue the following command which will give you back the public ip address of that instance.

   curl http://169.254.169.254/latest/meta-data/public-ipv4
   

Also take a look at this AWS documentation page on how to view web interfaces which provides other options like setting up SSH tunneling and using SOCKS proxy.