Oozie job is stuck in the Running state

2
votes

I have a simple job workflow which executes a mapreduce job as a shell action. After submitting the job, its status becomes Running and it stays there but never ends. The mapreduce cluster shows that there are two jobs running, one belongs to the shell application launcher and one for the actual mapreduce job. However the one for the mapreduce job is shown as UNASSIGNED and the progress is zero (which means it has been started yet).

Interestingly when I kill the oozie job, the mapreduce job actually starts running and completes successfully. It looks like the shell launcher is blocking it.

p.s. It is a simple workflow and there is no start or end date that may cause it wait.

2
hadoopmapreduceyarnoozieoozie-coordinator

2 Answers

0
votes

Please consider the below case as per you memory resource

Number of container are dependent on the number of blocksize. if you have 2 GB data of 512 mb block size, Yarn creates 4 maps and 1 reduce. While running the mapreduce we should follow some rules to submit the mapreduce job.(this should be applicable for small cluster)

You should configure the below property asper you RAM DISK and CORES.

<property>
    <description>The minimum allocation for every container request at the RM,
    in MBs. Memory requests lower than this won't take effect,
    and the specified value will get allocated at minimum.</description>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>512</value>
  </property>

  <property>
    <description>The maximum allocation for every container request at the RM,
    in MBs. Memory requests higher than this won't take effect,
    and will get capped to this value.</description>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>2048</value>
  </property>


 <property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>2048</value>
 </property>

And set the Java heap size as per the Memory Resources. Once ensured with the above property in yarn-site.xml according the mapreduce will succeed efficiently.

0
votes

When a job stuck in "UNASSIGNED" state, it usually means resource manager(RM) can't allocate container to the job. Check the capacity configure for the user and queue. Giving them more capacity should help.

With Hadoop 2.7 and capacity scheduler, specifically, the following properties need to be examined:

yarn.scheduler.capacity.<queue-path>.capacity
yarn.scheduler.capacity.<queue-path>.user-limit-factor
yarn.scheduler.capacity.maximum-applications 
  / yarn.scheduler.capacity.<queue-path>.maximum-applications
yarn.scheduler.capacity.maximum-am-resource-percent 
  / yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent

see more details on those properties at Hadoop: Capacity Scheduler - Queue Properties