Can't access HDFS when Hadoop cluster kerberized

0
votes

I successfully kerberized a test Hortonworks cluster. Ambari created keytabs for the services and they are all started. There is HA for namenodes. Standby namenode starts fast, the Active namenode takes much longer. Namenode UI shows that everything is correct. Can login by using kerberos. Namenodes are nn1.zim.com and nn2.zim.com

What can be wrong with this configuration? Login as hdfs, load keytab with kinit -kt. On list HDFS attempt I get this error:

[root@nn1 hdfs]# hdfs dfs -ls / 18/12/02 16:18:22 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSE xception: No valid credentials provided (Mechanism level: Failed to find any Ker beros tgt)] 18/12/02 16:18:22 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechani sm level: Failed to find any Kerberos tgt)]; Host Details : local host is: "nn1. zim.com/192.168.50.10"; destination host is: "nn2.zim.com":8020; , while invokin g ClientNamenodeProtocolTranslatorPB.getFileInfo over nn2.zim.com/192.168.50.11: 8020 after 1 failover attempts. Trying to failover after sleeping for 1123ms.

Kerberos principal for hosts are:

nn1.zim.com/[email protected] [email protected] nn2.zim.com/[email protected] [email protected] host/[email protected] host/[email protected]

The krb5.cfg:

[logging] default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults] dns_lookup_realm = false
ticket_lifetime = 24h renew_lifetime = 7d
forwardable = true
rdns = false default_realm = ZIM.COM
default_ccache_name = KEYRING:persistent:%{uid}

[realms] ZIM.COM = {
kdc = kb.zim.com
admin_server = kb.zim.com
}

[domain_realm]
.zim.com = ZIM.COM
zim.com = ZIM.COM

1
hadoophdfskerberosambari
Back to basics: log in as a Linux user and create a Kerberos ticket for user principal e.g. kinit [email protected] – Samson Scharfrichter
The default Kerberos credentials cache is /tmp/krb5cc_<id-of-current-user> so don't mess around with root or any other system / shared account. – Samson Scharfrichter
Recommended reading: "Hadoop and Kerberos, the Madness beyond the Gate" by Steve Loughran (HortonWorks). But read about Kerberos basics first. – Samson Scharfrichter
I've updated the question with the solution. – Oleg Hmelnits
better add solution as answer – Alexan

1 Answers

1
votes

SOLUTION: The two kerberos principals for each host have to be created: FQDN and short. I've created only FQDN (nn1.zim.com) - this was the cause of the issue. After creation of the second principal (nn1), everything started to work.

When you work with Active Directory, both types of the principals would be created automatically on AD Computer object creation.