Google Deployment Manager - BigTable example

Question

I have been trying this example provided in the Google's Deployment Manager GitHub project.

It works, yet I am not sure what is the purpose of creating three instances named instance_create, instance_update and instance_delete.

For example, taken from the link:

instance_create = {
      'name':
          'instance_create',
      'action':
          'gcp-types/bigtableadmin-v2:bigtableadmin.projects.instances.create',
      'properties': {
          'parent': project_path,
          'instanceId': instance_name,
          'clusters': copy.deepcopy(initial_cluster),
          'instance': context.properties['instance']
      },
      'metadata': {
          'runtimePolicy': ['CREATE']
      }
}

What is the purpose of `action` and `metadata`.`runtimePolicy`? I have tried to find it in the documentation but failed miserably. Why there are three `BigTable` instances there?

Joan Grau Noël Joan Grau Noël · Accepted Answer · 2019-01-10T14:12:07

You are right, the documentation is missing the information, which would answer your questions regarding these parameters.

However, it helps knowing what's going on in the Depoyment Manager example you linked.

First of all, the following line in the config.yaml is where the things get tricky:

resources:
- name: my-bigtable
  type: bigtable.py

This line will do a call to the bigtable.py python file, which sets the resource type of the deployment to that which are in it, under the GenerateConfig function. See how this is done here.

The resources are returned as {'resources': resources} at the end of it, being the resources variable a list of templates created there.

These templates have different name identifiers, which are set by the "name" tag. So you are not creating three different instances with the name of instance_create, instance_update and instance_delete in this file, but you are creating three templates with those names, that will later be appended to the resources list, and later returned to the config.yaml resources.type tag. These templates then will be sequentially build and executed by the deployment manager, once the create command is used. Note that they might appear out of order, this is due not using a schema.

It's easier to see this structure in a .yaml file format, for example, built with jinja, the template you posted would be:

resources:
- action: gcp-types/bigtableadmin-v2:bigtableadmin.projects.instances.create
  name: instance_create
  metadata:
    runtimePolicy:
    - CREATE
  properties:
    clusters:
      initial:
        defaultStorageType: HDD
        location: projects/<PROJECT_ID>/locations/<PROJECT_LOCATION>
        serveNodes: 4
    instance:
      displayName: My BigTable Instance.
      type: PRODUCTION
    instanceId: my-instance
    parent: projects/<PROJECT_ID>

Notice that the parameters under properties are the fields in the request body to bigtableadmin.projects.instances.create (which is nesting a clusters object parameters and a instance object parameters). Note that the InstanceId under properties is always the same, hence the BigTable instance, on which the templates do the calls, is always the same one.

The thing is that, not only the example you linked creates various templates to be run in the same script, but that the resource type for each template is a call to the BigTable API.

Normally the template resources are specified with the type tag, but since you are calling a resource that is directly running an API call (i.e. instead of just specifying gcp-types/bigtableadmin-v2, you are specifying bigtableadmin-v2:bigtableadmin.projects.instances.create), the action tag is used. I haven't found this difference on usage documented anywhere, but it needs to be specified like that. You will know if you are calling an API 'endpoint' directly if the resource ends with either create/update/delete.

Finally, the I have investigated in my side, and the metadata.runtimePolicy is tied to the fact that the resource type is an API call (like in the previous point). Once again, I haven't found this documented anywhere. However, since this is a requirement, you will always have to set the correct value in this field. It basically boils down to have metadata.runtimePolicy set to this values, depending on which type of API call you do:

create -> ['CREATE']

update -> ['UPDATE_ON_CHANGE']

delete -> ['DELETE']

Summarizing:

You are not creating three different instances, but three different templates, which do the work on the same BigTable instance.
You need to change the resource type flag to action if you are calling an API endpoint (create/update/delete), instead of just naming the base API.
The metadata.runtimePolicy value is a requirement when doing a call to one of the aforenamed endpoints.

Google Deployment Manager - BigTable example

1 Answers