Hadoop user/folder permissions

Question

I want to create directory for each user.

I looked at several how-to's and they say different things, I want it to be as easy as possible (I don't care about the encryption, as users will login to the machine using their ssh keys)

I've found this small guide: hadoop user file permissions

But have few questions,

1. Do I need to create directories and users on each slave/node machine too?

2. What is /user/myuser folder exactly? Is it supposed to be the /opt/hadoop/dfs/name/data (dfs.data.dir) folder in the $HADOOP_HOME/etc/hadoop/hdfs-site.xml file?

3. Do I also need to give/create a dfs.name.dir dir for each user?

After I create the users and directory, do I need to put some params in user's .bashrc file or give them specific permissions to use the hadoop commands? (put/delete files for example, create dirs...)

Anything else I forgot?

P.S My Hadoop works with sparks, if that matters.

Answer 1

Do I need to create the folders and users on each slave/node machine too?

No. It is enough to create the folders on the master either through a program or simple using hadoop fs -mkdir /foo

what is this /user/myuser folder exactly? is it supposed to be the /opt/hadoop/dfs/name/data ( dfs.data.dir )folder in the $HADOOP_HOME/etc/hadoop/hdfs-site.xml file?

The folder is what you'd expect of a standard users directory under home on linux. The user running the job/task/program has permissions in their folder. Note that these directories are not created by default by HDFS unless the users are added using something like Apache Ambari or Hue.

Do I also need to give/create a dfs.name.dir dir for each user?

You do not! They all share the same dfs