Launching a Spark-Submit job under Kerberos for Kafka

1
votes

Through tinkering I have been able to partially launch a spark submit job using the following command, however soon after starting it crashes and gives me the exception outlined below:

Spark-Submit Command:

su spark -c 'export SPARK_MAJOR_VERSION=2; spark-submit \ --verbose \ --master yarn \ --driver-cores 5 \ --num-executors 3 --executor-cores 6 \ --principal [email protected] \ --keytab /etc/security/keytabs/spark.headless.keytab \ --driver-java-options "-Djava.security.auth.login.config=kafka_client_jaas.conf"\ --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \ --files "/tmp/kafka_client_jaas.conf,/tmp/kafka.service.keytab" \ --class au.com.XXX.XXX.spark.test.test test.jar application.properties'

EXCEPTION:

Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user

WARN KerberosLogin: [Principal=kafka/[email protected]]: TGT renewal thread has been interrupted and will exit.

How can I get Kerberos to KINIT two principals at the same time? I'm assuming that is the problem here? I have tried adding another set of --principal/--keytab to the initial command, although this presented more permission issues within HDFS.

1
hadoopapache-sparkapache-kafkakerberoshortonworks-data-platform
I am facing same issue. Any solution to it ? - Elvish_Blade

1 Answers

0
votes

It's an old thread, but I struggled with this for some time and hopefully this can help someone.

The possible cause is that the Spark executors are not being able to locate the keytab, so they are failling to authenticate to Kerberos. On your submit, you should pass your Jaas config and Keytab files to your executors using the following options:

spark-submit --master yarn --deploy-mode cluster --files /path/to/keytab/yourkeytab.keytab#yourkeytab.keytab,/path/to/jaas/your-kafka-jaas.conf#your-kafka-jaas.conf --conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=your-kafka-jaas.conf" --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=your-kafka-jaas.conf" --driver-java-options "-Djava.security.auth.login.config=your-kafka-jaas.conf" your-application.jar

Finally, since these jaas files are being sent to executors (and the spark driver), you should use the relative path for the Keytab, and not the absolute. Your jaas config then should have the following line:

keyTab="./yourkeytab.keytab"