GCP Cloud Functions config file

Question

Let's say I want to create a simple cloud function to run a python script, where the main.py is in a github repository mirrored via Cloud Source Repositories. My questions is, if I need to reference information that I don't want to add to the repository - is there another way to access that information? For example, let's say I want to have a config.py which I reference in main.py. Is it possible to save and reference config.py somewhere in GCP instead? (e.g. Storage)?

Thanks!

Answer 1

Another answer that came to mind is the use of GCP's Runtime Configurator. This is an API within the Google Cloud Platform that lets you store information to use within GCE resources, e.g. cloud functions. Note that as we speak, this feature is still in Beta! Here is a small demo:

Create your project config:

gcloud beta runtime-config configs create my-project-config

Set a variable in your project config:

gcloud beta runtime-config configs variables set --config-name my-project-config --is-text my-variable "hello world"

The service account running the cloud function needs the following permissions:

runtimeconfig.configs.get
runtimeconfig.variables.list

Use that variable in a cloud function (Python):

from google.cloud import runtimeconfig

client = runtimeconfig.Client()
config = client.config('my-config')

print(config.get_variable('variable-name'))
# <Variable: my-config, variable-name>

print(config.get_variable('does-not-exist'))
# None

Answer 2

It seems like what you might want is Environment Variables for Cloud Functions or possibly even Secrets in Cloud Functions.

Other than that, Cloud Functions are completely stateless, so you'd need to connect to some external datastore like a database to load private configuration.

Answer 3

Look into variable substitution in Cloud Build where a 'build trigger' would contain non-repository values that would be inserted in 'build steps' into your Cloud Function as environment variables.

• https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values
• https://cloud.google.com/functions/docs/env-var

Answer 4

In addition to the other answers, we use a somewhat different approach. It boils down in having a public repo which contains all the cloud function Python code. We have another private repository which only contains configuration, like config.py. Let's describe this as an example:

1. Create 2 repositories, for example:

   github.com/organization/cloud-function (public)
   github.com/organization/config         (private)
   

2. Set a cloudbuild trigger on the config repository, and set a cloudbuild trigger on the cloud-function repository to trigger the build on the config repository. Here is some documentation about creating cloudbuild triggers.

3. In the last step everything comes together. Remember, your configuration is private, so not accessible to anyone else. Everytime someone pushes changes to one of the repositories, it should trigger the cloudbuild.yaml in your private repo. That cloudbuild.yaml looks something like this:

   ---
   timeout: 1800s
   steps:
     # Clone public repo
     - name: 'gcr.io/cloud-builders/git'
       args:
         - 'clone'
         - 'https://github.com/organization/cloud-function.git'
   
     # Copy config
     - name: 'gcr.io/cloud-builders/gcloud'
       entrypoint: 'bash'
       args:
         - '-c'
         - |
           cp config.py cloud-function/
   
     # Deploy cloud-function
     - name: 'gcr.io/cloud-builders/gcloud'
       dir: 'cloud-function'
       entrypoint: 'bash'
       args:
         - '-c'
         - |
           gcloud functions deploy ...
   

4. In addition, you can put references (secret_id) to Google Secret Manager secrets in your config.py. You could also use --env-vars-file for which the actual file is stored in the private repository. Another bonus is that you can have directories in your private repo which represent a $BRANCH_NAME or $PROJECT_ID, which makes it easy to create multiple environments (test, development, production etc.). This way you are sure the correct configuration for the environment is injected in the cloud function. We use this as follows:

   my-dev-gcp-project > build trigger on development branch
   my-prd-gcp-project > build trigger on production branch
   

   In the cloudbuild.yaml we clone the public repo with ${BRANCH_NAME} and copy the config from a source directory called ${PROJECT_ID}/config.py. With this setup you have clear separation between development and production config and code.