got Exec Failure: HTTP 403 when submit spark on k8s

Question

spark version: v2.4.0
eks info: v1.10.11-eks

after submit, got wrong message as follow:

019-02-21 15:08:44 WARN WatchConnectionManager:185 - Exec Failure: HTTP 403, Status: 403 - pods is forbidden: User "system:anonymous" cannot watch pods in the namespace "spark" java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: pods is forbidden: User "system:anonymous" cannot watch pods in the namespace "spark"

You need to grant access to the specified user into the namespace or run the process as a new user who has that access. — ferrix

coolinuxoid coolinuxoid · Accepted Answer · 2019-02-21T12:43:59

You need to create Role for system:anonymous user to watch pods on your namespace with similar to below yaml

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: spark # your namespace
  name: pod-reader # Role name will be needed for binding to user
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

Then you need to create RoleBindging to bind this role to system:anonymous user with similar to below yaml

# This role binding allows "system:anonymous" to read pods in the "spark" namespace.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: spark # your namespace
subjects:
- kind: User
  name: system:anonymous # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role #this must be Role or ClusterRole
  name: pod-reader # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

Documentation for more info about Anonymous requests

got Exec Failure: HTTP 403 when submit spark on k8s

1 Answers