Kubernetes Service Account for Jobs

Question

I have a requirement to invoke a Kubernetes job from another pod in the same namespace in a RBAC enabled cluster. The job is created using the C# Kubernetes client library. This task works correctly on a test environment that does not have RBAC enabled.

After following the Kubernetes documentation for creating a Service Account, I have the following YAML.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: x20-jobs-sp
  namespace: prod
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: prod
  name: x20-jobs-sp-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["batch", "extensions"]
  resources: ["jobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: x20-jobs-sp-rolebinding
  namespace: prod
subjects:
  - kind: ServiceAccount
    # Reference to ServiceAccount kind's `metadata.name`
    name: x20-jobs-sp
    # Reference to ServiceAccount kind's `metadata.namespace`
    namespace: prod
roleRef:
  kind: ClusterRole
  name: x20-jobs-sp-role
  apiGroup: rbac.authorization.k8s.io

This is the code that invokes the job.

var job = await client.CreateNamespacedJobAsync(new k8s.Models.V1Job
{
    Metadata = new k8s.Models.V1ObjectMeta
    {
        Name = safeName
    },
    Spec = new k8s.Models.V1JobSpec
    {
        Template = new k8s.Models.V1PodTemplateSpec
        {
            Spec = new k8s.Models.V1PodSpec
            {
                Containers = new List<k8s.Models.V1Container>()
                {
                    new k8s.Models.V1Container
                    {
                        Image = $"{_containerRegistry}/{jobName}:{_imageTag}",
                        Args = args.Select(x => x.ToString()).ToList(),
                        Env = GetDefaultEnvironment(),
                        Name = safeName,
                        ImagePullPolicy = "Always"  
                    }
                },
                ImagePullSecrets = new List<k8s.Models.V1LocalObjectReference>
                {
                    new k8s.Models.V1LocalObjectReference
                    {
                        Name = _containerRegistry
                    }
                },
                RestartPolicy = "OnFailure", 
                ServiceAccountName = "x20-jobs-sp" // jobs service principal
            }
        },
        TtlSecondsAfterFinished = _ttlSecondsAfterFinished
    }
}, _namespace);

This correctly create's the ServiceAccount, ClusterRole and ClusterRoleBinding, however attempting to invoke the task still result's in a Forbidden result.

I've attempted a number of different options for the ClusterRole including using the built in cluster-admin role, all without success. This has been tested using the kubectl auth can-i command.

What am I missing?

What is the specific error you get? It usually shows which version you are trying to use but don't have. — coderanger
What's your kubernetes version? Is bare metal installation or are you using some cloud provider? — Mr.KoopaKiller
Could you post an app as example or tell how we reproduce your scenario? Post the entire log from the request, this could help. — Mr.KoopaKiller

Brett G Brett G · Accepted Answer · 2020-02-10T01:34:25

So it turned out that the cause was the default service account for the pod that was attempting to invoke the job didn't have the necessary rolebindings.
I tested this by adding the cluster-admin RoleBinding to the default service account in the namespace and invoking the job.
kubectl create clusterrolebinding --user system:serviceaccount:prod:default prod-cluster-admin --clusterrole cluster-admin

Kubernetes Service Account for Jobs

1 Answers