pig script hang forever at heart beat

Question

I had installed all components of cloudera 5 in one machine: name node, datanode, hue, pig, oozie , yarn, hbase ...

I run the following pig script in shell:

sudo -u hdfs pig

and then in the pig shell run

data = LOAD '/user/test/text.txt' as (text:CHARARRAY) ;

DUMP data;

script work well

But when run this script on hue browsers Query Editor/ Pig Editor it stuck and below is the log:

2015-09-14 14:07:06,847 [uber-SubtaskRunner] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - More information at: http://HadoopTestEnv:50030/jobdetails.jsp?jobid=job_1442214247855_0002
2015-09-14 14:07:06,884 [uber-SubtaskRunner] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - 0% complete
2015-09-14 14:07:07,512 [communication thread] INFO  org.apache.hadoop.mapred.TaskAttemptListenerImpl  - Progress of TaskAttempt attempt_1442214247855_0001_m_000000_0 is : 1.0
Heart beat
2015-09-14 14:07:37,545 [communication thread] INFO  org.apache.hadoop.mapred.TaskAttemptListenerImpl  - Progress of TaskAttempt attempt_1442214247855_0001_m_000000_0 is : 1.0
Heart beat
2015-09-14 14:08:07,571 [communication thread] INFO  org.apache.hadoop.mapred.TaskAttemptListenerImpl  - Progress of TaskAttempt attempt_1442214247855_0001_m_000000_0 is : 1.0
Heart beat

I have using the yarn-utils script to support me config the yarn-site.xml and marped-site.xml

python yarn-ulti.spy -c 6 -m 16 -d 1 -k True
 Using cores=4 memory=16GB disks=1 hbase=True
 Profile: cores=6 memory=12288MB reserved=4GB usableMem=12GB disks=1
 Num Container=3
 Container Ram=4096MB
 Used Ram=12GB
 Unused Ram=4GB
 yarn.scheduler.minimum-allocation-mb=4096
 yarn.scheduler.maximum-allocation-mb=12288
 yarn.nodemanager.resource.memory-mb=12288
 mapreduce.map.memory.mb=2048
 mapreduce.map.java.opts=-Xmx1638m
 mapreduce.reduce.memory.mb=4096
 mapreduce.reduce.java.opts=-Xmx3276m
 yarn.app.mapreduce.am.resource.mb=2048
 yarn.app.mapreduce.am.command-opts=-Xmx1638m
 mapreduce.task.io.sort.mb=819

Script still hang and heart beat forever , any one help me please !

Here is my config: yarn-site.xml

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
        <value>12288</value>
          </property>

<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>6</value>
          </property>


  <property>
      <name>yarn.scheduler.minimum-allocation-mb</name>
          <value>4096</value>
            </property>

  <property>
      <name>yarn.scheduler.maximum-allocation-mb</name>
          <value>12288</value>
            </property>

<property>
      <name>yarn.app.mapreduce.am.resource.mb</name>
          <value>2048</value>
            </property>

<property>
      <name>yarn.app.mapreduce.am.command-opts</name>
          <value>-Xmx1638m</value>
            </property>

Mapred-site.xml

<property>
        <name>yarn.app.mapreduce.am.resource.mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.command-opts</name>
        <value>-Xmx768m</value>
    </property>


 <property>
     <name>yarn.app.mapreduce.am.staging-dir</name>
         <value>/user</value>
          </property>

<property>
    <name>mapreduce.map.java.opts</name>
        <value>-Xmx1638m</value>
          </property>

  <property>
      <name>mapreduce.reduce.java.opts</name>
          <value>-Xmx3276m</value>
            </property>

  <property>
      <name>mapreduce.map.memory.mb</name>
          <value>2048</value>
            </property>

  <property>
      <name>mapreduce.reduce.memory.mb</name>
          <value>4096</value>
            </property>


<property>
      <name>mapreduce.task.io.sort.mb</name>
          <value>819</value>
            </property>

<property>
    <name>mapreduce.map.cpu.vcores</name>
            <value>2</value>
                      </property>

<property>
    <name>mapreduce.reduce.cpu.vcores</name>
                <value>2</value>
 </property>

Romain Romain · Accepted Answer · 2015-09-14T23:55:21

The Pig App submits an Oozie jobs that will use one MR slot in addition to what the script does.

The locking is usually due to submission deadlocks like gotcha #5 or having only one available task slot in your cluster.

pig script hang forever at heart beat

1 Answers