Helm pre-install job is created before service account

2
votes

As part of my applications helm chart I have a Job which runs database migrations. I’ve annotated the job with the hook "helm.sh/hook": pre-install,pre-upgrade to ensure migrations are ran before the application is deployed. I want to use the same service account and config-map that my application deployment uses, however these resources have not been created at the time the job is executed resulting in the following error:

Warning FailedCreate 8s job-controller Error creating: pods "db-migrate-" is forbidden: error looking up service account dev-platform/platform: serviceaccount "platform" not found

According to the helm installation order the service account and config map should be created before the job. Is the behaviour nullified when running the job as pre-install?:

apiVersion: batch/v1
kind: Job
metadata:
  namespace: dev-platform
  name: db-migrate
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  activeDeadlineSeconds: 300
  backoffLimit: 1
  template:
    spec:
      # Share platform service account IAM role.
      serviceAccountName: {{ .Release.Name }}
      securityContext:
        fsGroup: 65534 #  Allow read permissions of AWS token files for IAM service account token.
      restartPolicy: Never
      containers:
        - name: db-migrate
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          envFrom:
            - configMapRef:
                name: platform-config
            - secretRef:
                name: platform-secrets
          # Overwrite APP_COMMAND variable.
          env:
            - name: APP_COMMAND
              value: migrate
            - name: APP_ENVIRONMENT
              value: {{ .Values.image.appEnvironment | quote }}
3
kuberneteskubernetes-helm

3 Answers

5
votes

Helm hooks are not limited to Jobs.

You can create the serviceaccount and configmap in the pre-install phase itself using the same helm-hook annotation as the Job.

Note: If you need the serviceaccount and configmap available after the pre-install phase, do not set the 'helm.sh/hook-delete-policy' to 'hook-succeeded'.

Example:

apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: my-sa    
    annotations:
      helm.sh/hook: pre-install
      helm.sh/hook-delete-policy: before-hook-creation
      helm.sh/hook-weight: "-10"
2
votes

Yes, From the official docs, a pre-install hook:

Executes after templates are rendered, but before any resources are created in Kubernetes (docs)

I would suggest to have the migrations as an init-container to your main app's pod. That way, reusing the existing configmaps and service accounts becomes trivial. Init-Containers need to run to completion before the pods containers are started. That way, you can also make sure that the database is migrated prior to starting your app. See here for the official documentation on init-containers.

0
votes

When I'm run ServiceAccount creation using Helm hooks as in the previous example getting this error (confusing a little):

Error: ServiceAccount "demo-33-service-account" is invalid: metadata.labels: Invalid value: "-10": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

It's because of helm.sh/hook-weight: "-10".

I'm also tried to use only positive weights (0 for ServiceAccount and 10 for Job), but with no luck - Job started before ServiceAccount created and stucked. When I manually remove the stucked Job, ServiceAccount created by hook successfully. Think it is because hooks executes after templates are rendered, but before any resources are created in Kubernetes (see pre-install in docs).

Maybe post-install hook for Job will rescue, but I solved the issue using initContainers for my migration job (thanks @winston).