historyserver not able to read log after enabling kerberos

0
votes

I enable the Kerberos on the cluster and it is working fine. But due to some issue mapred user is not able to read and display log over JobHistory server. I check the logs of job history server and it giving access error as:

org.apache.hadoop.security.AccessControlException: Permission denied:user=mapred, access=READ_EXECUTE, inode="/user/history/done_intermediate/prakul":prakul:hadoop:drwxrwx---

as we can see the directory have access to hadoop group and mapred is in hadoop group, even then it is not able to read the logs. Similar error it is giving for /tmp/logs/ folder due to which no log was displayed on resource manager UI.

I verify over all machine that hadoop group contains mapred user on all machine:

[cloudera]# id mapred uid=491(mapred) gid=489(mapred) groups=489(mapred),496(hadoop)

I also kinit the mapred user and try to access manually to these directory, but mapred not able to access even when folder having 770 permission:

[root@mn0 cloudera]# hdfs dfs -ls /tmp/logs/prakul
ls: Permission denied: user=mapred, access=READ_EXECUTE, inode="/tmp/logs/prakul":prakul:hadoop:drwxrwx---

[root@mn0 cloudera]# hdfs dfs -ls /tmp/logs/
Found 8 items
drwxrwx--- - xyz hadoop 0 2016-06-14 19:19 /tmp/logs/xyz
drwxrwx--- - abc hadoop 0 2016-06-13 06:06 /tmp/logs/abc
drwxrwx--- - prakul hadoop 0 2016-06-10 04:47 /tmp/logs/prakul

[root@mn0 cloudera]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: mapred/[email protected]
Valid starting Expires Service principal
06/27/16 01:07:32 06/27/16 11:07:32 krbtgt/[email protected]
renew until 07/04/16 01:07:32

If i give 777 permission to the directory then mapred is able to read and show log over UI as well as CLI.

Can any one know whether it is some Cloudera bug or there is some configuration issue due to which mapred not able to access the log even having full permission at group level?

I am using Cloudera 5.7 with Kerberos enabled.

thanks in advance

2
hadoophadoop-yarnkerberosclouderabigdata
Just to be sure that "mapred is in hadoop group": get a Kerberos ticket for mapred then run hdfs groups command. If you don't see "hadoop" in there, inspect your configuration about Group Mapping rules. – Samson Scharfrichter
Thanks Samson, I check that and found that mapred is not part of any group. so i add a hadoop group in my AD server and make hdfs, yarn and mapred member of that group. After that I am able to view the logs. Thanks again – Prakul Singhal

2 Answers

0
votes

I added a hadoop group in my AD server and made hdfs, yarn and mapred member of that group. After that I am able to view the logs.

0
votes

I have followed below permissions to cloudera /tmp/logs

/tmp/logs —————----

  1. Assume user1 is a valid local OS user. The following folder structure comprises of a proper JobHistory functionality:
 

    drwxrwxrwt - hdfs supergroup 0 2014-09-15 17:01 /tmp

    drwxrwxrwt - mapred hadoop 0 2014-09-18 12:02 /tmp/logs

    drwxrwx--- - user1 hadoop 0 2014-09-18 12:03 /tmp/logs/user1

    drwxrwx--- - user1 hadoop 0 2014-09-18 12:03 /tmp/logs/user1/logs

    
Here is an example entry for incorrect permissions:

    drwxrwx--- - hive supergroup 0 2014-09-18 12:00 /tmp/logs/user1/logs/


  2. Adjust the /tmp/logs/ folders recursively to reflect the ownership and permissions similar to the above:
 


Example commands to update the customer's permissions in HDFS:
 


sudo -u hdfs hadoop fs -chown mapred:hadoop /tmp/logs

sudo -u hdfs hadoop fs -chown -R :hadoop /tmp/logs/*