Hadoop DFS permission issue when running job

10
votes

I'm getting this following permission error, and am not sure why hadoop is trying to write to this particular folder:

hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar pi 2 100000
Number of Maps  = 2
Samples per Map = 100000
Wrote input for Map #0
Wrote input for Map #1
Starting Job
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=myuser, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

Any idea why it is trying to write to the root of my hdfs?

Update: After temporarily setting hdfs root (/) to be 777 permissions, I seen that a "/tmp" folder is being written. I suppose one option is to just create a "/tmp" folder with open permissions for all to write to, but it would be nice from a security standpoint if this is instead written to the user folder (i.e. /user/myuser/tmp)

4
permissionshadoop
Did you start hadoop with "myuser"? – Thomas Jungblut
No, I am using Cloudera, so hadoop is running as service, presumably as root – Dolan Antenucci
Try setting the permissions to / to 777 with hadoop fs -chmod and see what it writes. – Donald Miner
@orangeoctopus - it is writing /tmp/hadoop-mapred -- is there a setting to change this location? – Dolan Antenucci
@spike - I'd prefer it if non-root users can run hadoop from their own accounts. I noticed it is writing /tmp to hdfs, so trying to figure out why that is occurring, so let me know if you have any ideas on that. thanks – Dolan Antenucci

4 Answers

15
votes

I was able to get this working with the following setting:

<configuration>
    <property>
        <name>mapreduce.jobtracker.staging.root.dir</name>
        <value>/user</value>
    </property>

    #...

</configuration>

Restart of jobtracker service required as well (special thanks to Jeff on Hadoop mailing list for helping me track down problem!)

2
votes

1) Create the {mapred.system.dir}/mapred directory in hdfs using the following command

sudo -u hdfs hadoop fs -mkdir /hadoop/mapred/

2) Give permission to mapred user 

sudo -u hdfs hadoop fs -chown mapred:hadoop /hadoop/mapred/
0
votes

You can also make a new user named "hdfs". Quite simple solution but not as clean probably.

Of course this is when you are using Hue with Cloudera Hadoop Manager (CDH3)

0
votes

You need to set the permission for hadoop root directory (/) instead of setting the permission for the system's root directory. Even I was confused, but then realized that the directory mentioned was of hadoop's file system and not the system's.