Oozie java action gets killed then restarted by cluster

Question

I’m using an oozie java action step to start a java main. This java application does some calculations and then runs another map-reduce job based on that data. Since the oozie java action runs as a map-only job, it is also seen in job tracker.

One of our nodes was low on memory so the task tracker killed the oozie map-only job and restarted it on another node. However before killing it, the java application had already spawned its own map reduce job. When the oozie map-only job was restarted on the other node, it again spawned yet another map-reduce job with the same data as the former one. Looking in job tracker now has duplicate map-reduce jobs running against the same data.

How do you prevent/manage/alter settings such that the java program that oozie initiates in the map-only process only get run once, or is it necessary to have to constrain the Java application to be able to be run multiple times.

Any help would be appreciated,

Ken

Answer 1

There is not a lot you can do on the Oozie side if the one-mapper bootstrap jobs are failing because the hosts are out of memory. This host OOM scenario can be very problematic for every service in the cluster.

The preferred way to deal with this is to ensure that the host does not run out of memory at all by only allowing as many map and reduce slots on each TaskTracker node as there is memory available. You may also find that this allocation of resources to nodes is more efficient and tunable by using the YARN resource management framework instead of JobTracker-based MapReduce (MR1).