Apache Tez configuration with hadoop

Question

Here is what I have done in a nutshell:

STEP1: I have successfully configured hadoop 2.6 on my laptop (single node) and ran a sample mapreduce job.

STEP2: I cloned tez repository and successfully built the 0.8.0 version and copied the jarfiles into HDFS and exports the required variables. I also changed the value of variable mapreduce.framework.name to yarn-tez in the mapred-site.xml.

But when I want to run a tez orderedwordcount job, I got this error:

15/07/04 18:45:03 INFO ipc.Client: Retrying connect to server: hostname/hostIP:57339. 
Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/07/04 18:45:12 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED

I have checked resource manager and it is listening on port 8030. But it seems the client tries to connect to a random port. is it correct?

What can I do to get it work correctly?

Answer 1

It seems that it was the problem of this version (0.8.0) connecting to the resource manager. I compiled and integrated the previous stable release (0.7.0) and everything is good to go now. I hope that they will figure the problem out.

Answer 2

From your logs it seems a Firewall issue rather than issue with Tez version. And it is irrespective of Tez, even if you run Hadoop only you can face this.

Hadoop uses multiple ports for communication with clients and between service components. To enable Hadoop communication, open the specific ports that Hadoop uses. To open specific ports, you can set the access rules in Windows. For example, the following command will open up port 80 in the active Windows Firewall:

netsh advfirewall firewall add rule name=AllowRPCCommunication dir=in action=allow protocol=TCP localport=80

For more see here http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0-Win/bk_HDP_Install_Win/content/ref-79239257-778e-42a9-9059-d982d0c08885.1.html