management of kubernetes secrets

Question

we are starting with Kubernetes and wondering how other projects manage Kubernetes secrets:

• Since Kubernetes secrets values are just base64 encoded, it's not recommended to commit the secrets into source control
• If not committing to source control, it should be kept in some central place somewhere else, otherwise there's no single source of truth. If it's stored in some other places (.e.g. Hashicorp Vault), how's the integration with CI? Does the CI get values from Vault, create secrets resource on demand in Kubernetes?
• Another approach is probably to have a dedicated team to handle infrastructure and only that team knows and manages secrets. But if this team can potentially become a bottleneck if number of projects are large

Answer 1

how other projects manage Kubernetes secrets

Since they are not (at least not yet) proper secrets (base64 encoded), we treat them to separate restricted access git repository.

Most of our projects have code repository (with non-secret related manifests such as deployments and services as part of CI/CD pipeline) and separate manifest repository (holding namespaces, shared database inits, secrets and more or less anything that is either one-time init separate from CI/CD, requires additional permission to implement or that should be restricted in any other way such as secrets).

With that being said, although regular developer doesn't have access to restricted repository, special care must be given to CI/CD pipelines since even if you secure secrets, they are known (and can be displayed/misused) during CI/CD stage, so that can be weak security spot there. We mitigate that by having one of our DevOps supervise and approve (protected branches) any change to CI/CD pipeline in much the same manner that senior lead is supervising code changes to be deployed to production environment.

Note that this is highly dependent on project volume and staffing, as well as actual project needs in term of security/development pressure/infrastructure integration.

Answer 2

I came across this in github called SealedSecrets. https://github.com/bitnami-labs/sealed-secrets. I haven't used it myself. Though it seems to be a good alternative. But take note of this github issue (https://github.com/bitnami-labs/sealed-secrets/issues/92). It may cause you to lose labels and annotations.

In a nut shell SealedSecrets allows you to create a custom resource definition which encrypts your secret. In turn when you deploy the resource it will decrypt the CRD and turn it to a kubernetes Secret. This way you can commit your SealedSecret resource in your source code repo.

Answer 3

I use k8 secrets as the store where secrets are kept. As in when I define a secret I define it in k8 not somewhere else to then figure out how to inject it into k8. I have a handy client to create lookup and modify my secrets. I don't need to worry about my secrets leaving the firewall. They are easily injected into my services

If you want an extra layer of protection you can encrypt the secrets in k8 yourself with a KMS or something like that

Answer 4

We recently released a project called Kamus. The idea is to allow developers to encrypt secrets for a specific application (identified with a Kubernetes service account), while only this application can decrypt it. Kamus was designed to support GitOps flow, as the encrypted secret can be committed to source control. Take a look at this blog post for more details.