cdh4 hadoop-hbase PriviledgedActionException as:hdfs (auth:SIMPLE) cause:java.io.FileNotFoundException

4
votes

I have installed cloudera cdh4 release And I am trying to run the mapreduce job on that. I am getting following error -->

2012-07-09 15:41:16 ZooKeeperSaslClient [INFO] Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-07-09 15:41:16 ClientCnxn [INFO] Socket connection established to Cloudera/192.168.0.102:2181, initiating session
2012-07-09 15:41:16 RecoverableZooKeeper [WARN] Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
2012-07-09 15:41:16 RetryCounter [INFO] The 1 times to retry  after sleeping 2000 ms
2012-07-09 15:41:16 ClientCnxn [INFO] Session establishment complete on server Cloudera/192.168.0.102:2181, sessionid = 0x1386b0b44da000b, negotiated timeout = 60000
2012-07-09 15:41:18 TableOutputFormat [INFO] Created table instance for exact_custodian
2012-07-09 15:41:18 NativeCodeLoader [WARN] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2012-07-09 15:41:18 JobSubmitter [WARN] Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2012-07-09 15:41:18 JobSubmitter [INFO] Cleaning up the staging area file:/tmp/hadoop-hdfs/mapred/staging/hdfs48876562/.staging/job_local_0001
2012-07-09 15:41:18 UserGroupInformation [ERROR] PriviledgedActionException as:hdfs (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /home/cloudera/yogesh/lib/hbase.jar
Exception in thread "main" java.io.FileNotFoundException: File does not exist: /home/cloudera/yogesh/lib/hbase.jar
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:736)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:246)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:284)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:355)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244)
    at

I am able to run sample programs given in hadoop-mapreduce-examples-2.0.0-cdh4.0.0.jar. But I am getting this error when my job is submitted successfully to jobtracker . Looks like it is trying to access local file-system again (Although I have set all the required libraries for job execution in distributed cache still its trying to access local dir). Is this issues related to user privileges ?

I) Cloudera:~ # hadoop fs -ls hdfs://<MyClusterIP>:8020/ shows - 

Found 8 items
drwxr-xr-x   - hbase hbase               0 2012-07-04 17:58 hdfs://<MyClusterIP>:8020/hbase<br/>
drwxr-xr-x   - hdfs  supergroup          0 2012-07-05 16:21 hdfs://<MyClusterIP>:8020/input<br/>
drwxr-xr-x   - hdfs  supergroup          0 2012-07-05 16:21 hdfs://<MyClusterIP>:8020/output<br/>
drwxr-xr-x   - hdfs  supergroup          0 2012-07-06 16:03 hdfs:/<MyClusterIP>:8020/tools-lib<br/>
drwxr-xr-x   - hdfs  supergroup          0 2012-06-26 14:02 hdfs://<MyClusterIP>:8020/test<br/>
drwxrwxrwt   - hdfs  supergroup          0 2012-06-12 16:13 hdfs://<MyClusterIP>:8020/tmp<br/>
drwxr-xr-x   - hdfs  supergroup          0 2012-07-06 15:58 hdfs://<MyClusterIP>:8020/user<br/>

II) --- No Result for following ----

hdfs@Cloudera:/etc/hadoop/conf> find . -name '**' | xargs grep "default.name"<br/>
hdfs@Cloudera:/etc/hbase/conf> find . -name '**' | xargs grep "default.name"<br/>

Instead I think with new APIS we are using ->
fs.defaultFS --> hdfs://Cloudera:8020 which i have set properly

Although for "fs.default.name" I got entries for hadoop cluster 0.20.2 (non-cloudera cluster)

cass-hadoop@Pratapgad:~/hadoop/conf> find . -name '**' | xargs grep "default.name"<br/>
./core-default.xml:  <name>fs.default.name</name><br/>
./core-site.xml:  <name>fs.default.name</name><br/>

I think cdh4 default configuration should add this entry in respective directory. (If its bug).

The command I am using to run my progrmme -

hdfs@Cloudera:/home/cloudera/yogesh/lib> java -classpath hbase-tools.jar:hbase.jar:slf4j-log4j12-1.6.1.jar:slf4j-api-1.6.1.jar:protobuf-java-2.4.0a.jar:hadoop-common-2.0.0-cdh4.0.0.jar:hadoop-hdfs-2.0.0-cdh4.0.0.jar:hadoop-mapreduce-client-common-2.0.0-cdh4.0.0.jar:hadoop-mapreduce-client-core-2.0.0-cdh4.0.0.jar:log4j-1.2.16.jar:commons-logging-1.0.4.jar:commons-lang-2.5.jar:commons-lang3-3.1.jar:commons-cli-1.2.jar:commons-configuration-1.6.jar:guava-11.0.2.jar:google-collect-1.0-rc2.jar:google-collect-1.0-rc1.jar:hadoop-auth-2.0.0-cdh4.0.0.jar:hadoop-auth.jar:jackson.jar:avro-1.5.4.jar:hadoop-yarn-common-2.0.0-cdh4.0.0.jar:hadoop-yarn-api-2.0.0-cdh4.0.0.jar:hadoop-yarn-server-common-2.0.0-cdh4.0.0.jar:commons-httpclient-3.0.1.jar:commons-io-1.4.jar:zookeeper-3.3.2.jar:jdom.jar:joda-time-1.5.2.jar com.hbase.xyz.MyClassName

2
configurationhadoopmapreducehbasecloudera
Can you post the command line for you job submission, or any code which references this file - does the file exist on the local system? – Chris White
Hi Chris, Thanks for reply I have updated the question please see above. – Yogesh
job_local_0001 means mapred-site.xml is not set properly.And should be used in while using New Configuration(). set there. hbase.apache.org/book.html#trouble.mapreduce.local – Yogesh
@Yogesh, you should provide this as an answer to the question, as it was the important key that unlocked the issue for me. Once I set mapred-site.xml as described (somewhat unclearly and incompletely) by Ramy below, the problem was resolved. – barclay

2 Answers

2
votes

Debug procedure: Try running simple Hadoop shell commands.

hadoop fs -ls /

If this shows the HDFS files then your configuration is correct. If not, then the configuration is missing. When this happens hadoop shell command like -ls will refer to local filesystem and not Hadoop file system. This happens if Hadoop is started using CMS (Cloudera manager). It does not explicitly stores the configuration in conf directory.

Check if hadoop file system is displayed by following command (change port):

hadoop fs -ls hdfs://host:8020/

If it displays local file system when you submit the path as / then you should set the configuration files hdfs-site.xml and mapred-site.xml in configuration directory. Also hdfs-site.xml should have the entry for fs.default.name pointing to hdfs://host:port/. In my case the directory is /etc/hadoop/conf.

See: http://hadoop.apache.org/common/docs/r0.20.2/core-default.html

See, if this resolves your issue.

4
votes

Even I phased the same problem in 2.0.0-cdh4.1.3 while running MR jobs. After adding the property in mapred.site.xml

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

For running Hive job

export HIVE_USER=yarn