Kerberos Auth With SparkLauncher - "Cannot locate KDC" Error

0
votes

We are trying to move from using spark-submit to a programmatic solution using SparkLauncher (Spark 2.4 running on Cloudera Hadoop), and have encountered this problem:

Exception javax.security.auth.login.LoginException: Cannot locate KDC

on the error output stream of a job process after calling startApplication().

The launcher is configured with the same settings as spark-submit, including setting "spark.kerberos.principal" and "spark.kerberos.keytab" (also tried using the spark.yarn prefix, and setting --principle and --keytab with setSparkArg() as well - but these properties are not the issue).

Even though krb5.conf is in the default location on all machines and the driver, I tried setting

.setConf("spark.executor.extraJavaOptions", "-Djava.security.krb5.conf=/etc/krb5.conf")
.setConf("spark.driver.extraJavaOptions", "-Djava.security.krb5.conf=/etc/krb5.conf")

as well, with the same result.

There are no other auth/kerberos-related conf settings with the spark-submit call (only principal and keytab are set), and all other conf settings are replicated on the Launcher.

The output from SparkLauncher confirms all Spark properties are set as expected.

"Cannot locate KDC" indicates that the krb5.conf file is not found, but how can this be if it's in the default location and spark-submit evidently finds it?

1
scalaapache-sparkhadoopcloudera
"Cannot locate KDC" may not imply that krb5.conf is not there, but may be a configuration problem with krb5.conf. You can check whether krb5.conf across all machines have same configurations. If yes, you can try doing a manual authentication using keytab on any of the machine kinit <principal_name> -kt <path_to_keytab_file> and see how it goes. – Syam Sankar
The debug traces from -Djava.security.debug=gssloginconfig,configfile,configparser,logincontext might explain what happens exacty when JAAS tries to connect to the KDC. – Samson Scharfrichter
Try also -Dsun.security.krb5.debug=true if you are not afraid of huge amounts of cryptical stuff. – Samson Scharfrichter
@SyamSankar I know the krb5.confs are good because spark-submit, launched with k5start (similar to kinit), works. – user14744699
To be clear, since the same settings and same krb5 work with spark-submit, this almost has to come down to some difference between it and SparkLauncher. – user14744699

1 Answers

1
votes

This error does not necessarily mean that it cannot locate your krb5.conf file. It can also arise when your krb5.conf file does not contain the correct domain or realm specifications for your principal.

For example, if your principal is [email protected] but your krb5.conf file only specifies [email protected], you will also get this error.

Additionally, if your krb5.conf is located in a location other than default (/etc/krb5.conf), I was having issues with setting the javaExtraOpts. I found pointing updating the env var prior to running helped: export SPARK_SUBMIT_OPTS="-Djava.security.krb5.conf=/path/to/krb5-new.conf"

Lastly, I found setting --principle and --keytab args in spark-submit actually caused issues, and removing these can help.