can not set “spark” as service account name when run spark-submit command

Question

I am running this command in my mac terminal,want to submit my test spark job on to one of our k8s cluster:

ID_TOKEN=`kubectl config view --minify -o jsonpath='{.users[0].user.auth-provider.config.id-token}'`

./bin/spark-submit \
    --master k8s://https://c2.us-south.containers.cloud.ibm.com:30326 \
    --deploy-mode cluster \
    --name Hello \
    --class scala.example.Hello \
    --conf spark.kubernetes.namespace=isap \
    --conf spark.executor.instances=3 \
    --conf spark.kubernetes.container.image.pullPolicy=Always \
    --conf spark.kubernetes.container.image.pullSecrets=default-us-icr-io \
    --conf spark.kubernetes.container.image=us.icr.io/cedp-isap/spark-for-apps:2.4.1 \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.authenticate.driver.caCertFile=/usr/local/opt/spark/ca.crt \
    --conf spark.kubernetes.authenticate.submission.oauthToken=$ID_TOKEN \
    local:///opt/spark/jars/interimetl_2.11-1.0.jar

And I already created service account "spark", as well as cluster role binding yaml like this:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: isap
  name: pod-mgr
rules:
- apiGroups: ["rbac.authorization.k8s.io", ""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list", "create", "delete"]

and

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-mgr-spark
  namespace: isap
subjects:
- kind: ServiceAccount
  name: spark
  namespace: isap
roleRef:
  kind: ClusterRole
  name: pod-mgr
  apiGroup: rbac.authorization.k8s.io

But when I run above spark-submit command, I found the the log like this:

20/06/15 02:45:02 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
 pod name: hello-1592203500709-driver
 namespace: isap
 labels: spark-app-selector -> spark-0c7f50ab2d21427aac9cf2381cb4bb64, spark-role -> driver
 pod uid: 375674d2-784a-4b32-980d-953488c8a8b2
 creation time: 2020-06-15T06:45:02Z
 service account name: default
 volumes: kubernetes-credentials, spark-local-dir-1, spark-conf-volume, default-token-p8pgf
 node name: N/A
 start time: N/A
 container images: N/A
 phase: Pending
 status: []

You will notice it is still using service account "default", rather than "Spark" And the executor pod can not be created in my k8s cluster. Also no logs is displayed in created driver pod.

Could anyone can help to take a look what I missed here?Thanks!

Arghya Sadhu Arghya Sadhu · Accepted Answer · 2020-06-15T07:46:15

From the docs you need to use the service account spark as a parameter to spark submit

--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark

can not set “spark” as service account name when run spark-submit command

1 Answers