Problem deploying cockroachdb-client-secure

Question

I am following this helm + secure - guide:

https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html#helm

I deployed the cluster with this command: $ helm install my-release --values my-values.yaml cockroachdb/cockroachdb --namespace=thesis-crdb

This is how it looks: $ helm list --namespace=thesis-crdb

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                                      APP VERSION
my-release      thesis-crdb     1               2021-01-31 17:38:52.8102378 +0100 CET   deployed        cockroachdb-5.0.4                          20.2.4

Here is how it looks using: $ kubectl get all --namespace=thesis-crdb

NAME                                    READY   STATUS      RESTARTS   AGE
pod/my-release-cockroachdb-0            1/1     Running     0          7m35s
pod/my-release-cockroachdb-1            1/1     Running     0          7m35s
pod/my-release-cockroachdb-2            1/1     Running     0          7m35s
pod/my-release-cockroachdb-init-fhzdn   0/1     Completed   0          7m35s

NAME                                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)              AGE
service/my-release-cockroachdb          ClusterIP   None         <none>        26257/TCP,8080/TCP   7m35s
service/my-release-cockroachdb-public   ClusterIP   10.xx.xx.x   <none>        26257/TCP,8080/TCP   7m35s

NAME                                      READY   AGE
statefulset.apps/my-release-cockroachdb   3/3     7m35s

NAME                                    COMPLETIONS   DURATION   AGE
job.batch/my-release-cockroachdb-init   1/1           43s        7m36s

In the my-values.yaml-file I only changed the tls from false to true:

tls:
  enabled: true

So far so good, but from here on the guide isn't really working for me anymore. I try as they say with getting the csr: kubectl get csr --namespace=thesis-crdb

No resources found

Ok, perhaps not needed. I carry on to deploy the client-secure I download the file: https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/client-secure.yaml

And changes the serviceAccountName: cockroachdb to serviceAccountName: my-release-cockroachdb.

I try to deploy it with $ kubectl create -f client-secure.yaml --namespace=thesis-crdb but it throws this error:

Error from server (Forbidden): error when creating "client-secure.yaml": pods "cockroachdb-client-secure" is forbidden: error looking up service account thesis-crdb/my-release-cockroachdb: serviceaccount "my-release-cockroachdb" not found

Anyone got an idea how to solve this? I'm fairly sure it's something with the namespace that is messing it up.

I have tried to put the namespace in the metadata-section

metadata:
  namespace: thesis-crdb

And then try to deploy it with: kubectl create -f client-secure.yaml but to no avail:

Error from server (Forbidden): error when creating "client-secure.yaml": pods "cockroachdb-client-secure" is forbidden: error looking up service account thesis-crdb/my-release-cockroachdb: serviceaccount "my-release-cockroachdb" not found

Answer 1

You mention in question that you have changed serviceAccountName in YAML.

And changes the serviceAccountName: cockroachdb to serviceAccountName: my-release-cockroachdb.

So Root Cause of your issue is related with ServiceAccount misconfiguration.

Background

In your cluster you have something called ServiceAccount.

When you (a human) access the cluster (for example, using kubectl), you are authenticated by the apiserver as a particular User Account (currently this is usually admin, unless your cluster administrator has customized your cluster). Processes in containers inside pods can also contact the apiserver. When they do, they are authenticated as a particular Service Account (for example, default).

To ServiceAccount you also should configure RBAC which grants you permissions to create resources.

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization.

RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API.

If you don't have proper RBAC permissions you will not be able to create resources.

In Kubernetes you can find Role and ClusterRole. Role sets permissions within a particular namespace and ClusterRole sets permissions in whole cluster. Besides that, you also need to bind roles using RoleBinding and ClusterRoleBinding.

In addition, if you would use Cloud environment, you would also need special rights in project. Your guide provides instructions to do it here.

Root cause

I've checked cockroachdb chart and it creates ServiceAccount, Role, ClusterRole, RoleBinding and ClusterRoleBinding for cockroachdb and prometheus. There is no configuration for my-release-cockroachdb.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cockroachdb
...
  verbs:
  - create
  - get
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cockroachdb
  labels:
    app: cockroachdb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cockroachdb
...

In client-secure.yaml you change serviceAccountName to my-release-cockroachdb and Kubernetes cannot find that ServiceAccount as it was not created by cluster administrator or cockroachdb chart.

To list ServiceAccounts in default namespace you can use command $ kubectl get ServiceAccount, however if you would check all ServiceAccounts in cluster you should add -A to your command - $ kubectl get ServiceAccount -A.

Solution

Option 1 is to use existing ServiceAccount with proper permissions like SA created by cockroachdb chart which is cockroachdb, not my-release-cockroachdb.

Option 2 is to create ServiceAccount, Role/ClusterRole and RoleBinding/ClusterRoleBinding for my-release-cockroachdb.