Example map-reduce oozie program not working on CDH 4.5

Question

I am using Hadoop (CDH 4.5) on my mac since some time now, and do run map reduce jobs regularly. I installed oozie recently (again, CDH4.5) following instructions at: http://archive.cloudera.com/cdh4/cdh/4/oozie-3.3.2-cdh4.5.0/DG_QuickStart.html, and tried to run sample programs provided. However, it always fails with the following error. Looks like the workflow is not getting run at all. The Console URL field in the Job info is also empty.

Could someone please help on this?

The relevant snippet of the Oozie Job log follows.

2014-06-10 17:27:18,414  INFO ActionStartXCommand:539 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@:start:] Start action [0000000-140610172702069-oozie-usrX-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-06-10 17:27:18,417  WARN ActionStartXCommand:542 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@:start:] [***0000000-140610172702069-oozie-usrX-W@:start:***]Action status=DONE
2014-06-10 17:27:18,417  WARN ActionStartXCommand:542 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@:start:] [***0000000-140610172702069-oozie-usrX-W@:start:***]Action updated in DB!
2014-06-10 17:27:18,576  INFO ActionStartXCommand:539 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] Start action [0000000-140610172702069-oozie-usrX-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-06-10 17:27:19,188  WARN MapReduceActionExecutor:542 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] credentials is null for the action
2014-06-10 17:27:19,423  WARN ActionStartXCommand:542 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Unknown rpc kind RPC_WRITABLE]
org.apache.oozie.action.ActionExecutorException: JA009: Unknown rpc kind RPC_WRITABLE
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:773)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:927)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unknown rpc kind RPC_WRITABLE
at org.apache.hadoop.ipc.Client.call(Client.java:1238)
at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
at org.apache.hadoop.mapred.$Proxy30.getDelegationToken(Unknown Source)
at org.apache.hadoop.mapred.JobClient.getDelegationToken(JobClient.java:2125)
at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:372)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:970)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:723)
... 10 more

2014-06-10 17:27:19,426  INFO ActionStartXCommand:539 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] Next Retry, Attempt Number [1] in [60,000] milliseconds
2014-06-10 17:28:19,468  INFO ActionStartXCommand:539 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] Start action [0000000-140610172702069-oozie-usrX-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

Ramanan Ramanan · Accepted Answer · 2014-06-12T11:12:56

Have you set up hadoop with MapReduce or YARN?

Configuring which Hadoop Version to Use
The Oozie client does not interact directly with Hadoop MapReduce, and so it does not require any MapReduce configuration.

The Oozie server can work with either MRv1 or YARN. It cannot work with both simultaneously.

The MapReduce version Oozie server works with is determined by the CATALINA_BASE variable in the /etc/oozie/conf/oozie-env.sh file. By default, CATALINA_BASE is set to /usr/lib/oozie/oozie-server-0.20. This setting configures the Oozie server to work with MRv1.

To configure the Oozie server to work with YARN instead, set CATALINA_BASE to /usr/lib/oozie/oozie-server.

Since you have cdh4 distribution, install oozie following this link http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_17.html

If you have YARN,in your job.properties change port of jobtracker

nameNode=hdfs://localhost:8020 
jobTracker=localhost:8021

Instead of port 8021 (jobTracker), use resource manager's port: 8032

jobTracker=localhost:8032

https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/MlsqeUdMpgA

Example map-reduce oozie program not working on CDH 4.5

1 Answers