Oozie hive script scheduling in tez mode

Question

I tried to schedule the hive workflow xml file to run the hive script in tez mode by passing the hadoop properties for referring the tez jar files in workflow xml file as shown below.

            <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <job-xml>hive-site.xml</job-xml>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>tez.lib.uris</name>
                    <value>${nameNode}/apps/Tez/,${nameNode}/apps/Tez/lib/</value>     
                </property>
            </configuration>

Also I had changed the hive-site xml file property hive.execution.engine as tez mode.

           <property>
                <name>hive.execution.engine</name>
                <value>tez</value>      
           </property>

When I scheduled the workflow using oozie. I got the error as follows

    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, org/apache/tez/dag/api/SessionNotRunning
java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:479)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
    at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:306)
    at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:290)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
    at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:68)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.api.SessionNotRunning
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 24 more

Can any one please say how to rectify this issue and to schedule my workflow xml file and run the hive script in tez mode.

jc mannem jc mannem · Accepted Answer · 2016-04-06T13:43:11

I observed the above error before and I was able to resolve it and was able to run Hive(tez engine) on Oozie.

Here's the steps I followed.

Class not found error:

As the error says, Oozie Launcher container cannot find the SessionNotRunning class.

This class is part of tez-api-0.x.x.jar. you can confirm that using :

jar tvf /usr/lib/tez/tez-api-0.7.0.jar | grep SessionNotRunning

You need to make sure your Oozie launcher container(which is YARN) localize this and other TEZ JAR's for it to pass it to HiveClient.

The expectation is that if we include the following config property in the workflow.xml , oozie should pick up all those JAR's.

        <property>
            <name>tez.lib.uris</name>
            <value>hdfs:///apps/tez/,hdfs:///apps/tez/lib/</value>
        </property>

However, it may not do that.(Not sure why)

So, I copied all TEZ JAR's to Hive-action's share library in HDFS (Ex: to /user/oozie/share/lib/lib_20160405125827/hive/). The oozie hive-action on your workflow should use JAR's present in that path and localize those JAR's.

While doing that, make sure the new JAR's have same permission as previous JAR's present in that HDFS directory. Oozie also need a refresh of share library.

Example commands can be:

hadoop fs -copyFromLocal /usr/lib/tez/*.jar /user/oozie/share/lib/lib_20160405125827/hive/

hadoop fs -copyFromLocal /usr/lib/tez/lib/*.jar /user/oozie/share/lib/lib_20160405125827/hive/

hadoop fs -chown oozie:oozie /user/oozie/share/lib/lib_20160405125827/hive/*.jar

oozie admin -sharelibupdate

Now, if you list your hive share library, oozie admin -shareliblist hive , you should be able to see all TEZ libraries.

With those steps, you should no longer see NoClassDefFoundError's or ClassNotFoundException's from TEZ jars.

Missing Hadoop Dependencies:

At This time, the TEZ job should be submitted, but there's another error that you may encounter on the OOZIE launcher .

14972 [uber-SubtaskRunner] ERROR org.apache.hadoop.hive.ql.exec.Task - Failed to execute tez graph. org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1459860815404_0033 failed 2 times due to AM Container for appattempt_1459860815404_0033_000002 exited with exitCode: 1

looking at container logs, I see

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/service/AbstractService Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.AbstractService

This is because the my TEZ installation is minimal and doesn't come with hadoop dependencies.

https://github.com/apache/tez/blob/release-0.7.0/docs/src/site/markdown/install.md#hadoop-installation-dependent-installdeploy-instructions

So, you need to tell TEZ to use your cluster's hadoop libraries using the following property in your workflow.xml.

                         <property>
                              <name>tez.use.cluster.hadoop-libs</name>
                              <value>true</value>
                         </property>

So, with the above steps , I was able to run a hive script successfully on TEZ engine via Oozie.

Oozie hive script scheduling in tez mode

1 Answers