Access google cloud secret inside cloud build yaml

Question

I am trying to get the secrets from google secret manager. As per documentation to access the secret, the entrypoint should be bash, but I've different entrypoint. Trying to figure out how I can get the secret in my step 'Create dataflow template'. Seems environment variable value is not accessible outside step.

steps:
  - id: 'Pull dataflow-python3 docker image & load secrets'    
    name: 'gcr.io/$PROJECT_ID/dataflow-python3:latest'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        git rev-parse --short HEAD > COMMIT_ID
        commitId=$(cat COMMIT_ID)
        echo "Project Id is $PROJECT_ID"
        echo "Commit SHA is $COMMIT_SHA"
        echo "Commit Id is ${commitId}"
        echo "Secret key 'TEST' has value '$$TEST'"
    secretEnv: ['TEST']

  - id: "Activate virtual environment venv"
    name: 'gcr.io/$PROJECT_ID/dataflow-python3:latest'
    entrypoint: '/bin/bash'
    args: [ '-c', 'source /venv/bin/activate' ]
    waitFor: ['-']

  - id: "Create dataflow template"
    name: 'gcr.io/$PROJECT_ID/dataflow-python3:latest'
    entrypoint: 'python'
    args: 
    - -m
    - main
    - --job_name=test-df
    - --project=$PROJECT_ID
    - --region=us-east1
    - --template_location=gs://my-project-dataflow-templates/test-alerts-template/templates/send-alert-template
    - --staging_location=gs://my-project-dataflow-templates/test-alerts-template/staging/
    - --temp_location=gs://my-project-dataflow-templates/test-alerts-template/temp/
    - --runner=DataflowRunner
    - --setup_file='./setup.py'
    - --autoscaling_algorithm=NONE
    - --DUMMY=$$TEST
    secretEnv: ['TEST']
    waitFor: [
      'Pull dataflow-python3 docker image & load secrets',
      'Activate virtual environment venv'
      ]

availableSecrets:
  secretManager:
  - versionName: projects/my-project/secrets/TEST/versions/latest
    env: 'TEST'

Value passed in Dataflow Pipeline

Secret value:

Build Log 1:

Build Log 2:

Answer 1

I tested in different configuration and the behavior is "consistent" with the env parameter in Cloud Build. In fact you can't use env or secretEnv like a substitution variables.

The env and secretEnv are only available in the execution context of the command, not in the configuration of the command.

That means:

• if you run an app or a script which use environment variables, the env and secretEnv work.
• if you use the env or secretEnv in the args of a step, it doens't work.

---

But it's subtil

Here is the context of the the execution of BASH command -> Work

  - name: 'gcr.io/cloud-builders/gcloud'
    entrypoint: "bash"
    args:
      - -c
      - echo $$TEST
    secretEnv: ['TEST']

Here in argument of the step echo -> Don't work

  - name: 'gcr.io/cloud-builders/gcloud'
    entrypoint: "echo"
    args:
      - $$TEST
    secretEnv: ['TEST']

---

So to fix your problem, do this

  - id: "Create dataflow template"
    name: 'gcr.io/$PROJECT_ID/dataflow-python3:latest'
    entrypoint: 'bash'
    args: 
    - -c
    - python main \
     --job_name=test-df \
     --project=$PROJECT_ID \
     --region=us-east1 \
     --template_location=gs://my-project-dataflow-templates/test-alerts-template/templates/send-alert-template \
     --staging_location=gs://my-project-dataflow-templates/test-alerts-template/staging/ \
     --temp_location=gs://my-project-dataflow-templates/test-alerts-template/temp/ \
     --runner=DataflowRunner \
     --setup_file='./setup.py' \
     --autoscaling_algorithm=NONE \
     --DUMMY=$$TEST
    secretEnv: ['TEST']
    waitFor: [
      'Pull dataflow-python3 docker image & load secrets',
      'Activate virtual environment venv'
      ]

Answer 2

The code offered by @guillaume blaquiere will work, but the root cause of the issue is the entrypoint, as mentioned in Configuring builds to access the secret from Secret Manager:

In the build step where you want to specify the secret:

• Add an entrypoint field pointing to bash to use the bash tool in the build step. This is required to refer to the environment variable for the secret.
• Add a secretEnv field specifying the environment variable.
• In the args field, add a -c flag as the first argument. Any string you pass after -c is treated as a command. For more information on running bash commands with -c, see the bash documentation.
• When specifying the secret in the args field, specify it using the environment variable prefixed with $$.

In your example you use:

 entrypoint: 'python'
args: 
    - -m

The entrypoint is not bash and the first arg is not set the flag -c.

This also happens with the example of Guillaume:

 - name: 'gcr.io/cloud-builders/gcloud'
    entrypoint: "echo"
    args:
      - $$TEST
    secretEnv: ['TEST']

The entrypoint is not bash, but in his first example it is.

On the other hand, in the solution offered, the entry point is bash and the first arg is the flag -c, so this will work:

- id: "Create dataflow template"
    name: 'gcr.io/$PROJECT_ID/dataflow-python3:latest'
    entrypoint: 'bash'
    args: 
    - -c
    - python main \
     --job_name=test-df \
     --project=$PROJECT_ID \
     --region=us-east1 \
     --template_location=gs://my-project-dataflow-templates/test-alerts-template/templates/send-alert-template \
     --staging_location=gs://my-project-dataflow-templates/test-alerts-template/staging/ \
     --temp_location=gs://my-project-dataflow-templates/test-alerts-template/temp/ \
     --runner=DataflowRunner \
     --setup_file='./setup.py' \
     --autoscaling_algorithm=NONE \
     --DUMMY=$$TEST
    secretEnv: ['TEST']
    waitFor: [
      'Pull dataflow-python3 docker image & load secrets',
      'Activate virtual environment venv'
      ]

The solution is accurate but the reason is the entrypoint and the first flag -c