How to use the gRPC Python Plugin with Docker and Google Cloud Builds?

Question

TL,DR;

What Linux Docker image would be fastest / lightest to run the Python gRPC plugin when generating API descriptor files?
Should the aforementioned API descriptor become a Cloud Build artifact and saved to a Cloud Storage Bucket?
- ...in order to use gcloud to deploy the API to Cloud Endpoints.

Detail

I run a Python gRPC service and ESP in Docker containers running on Google Compute Engine. About gRPC > API management shows a diagram of my application architecture:

My high-level build steps:

1) Create the descriptor file, api_descriptor.pb, using the protoc protocol buffers compiler.

python -m grpc_tools.protoc \
    --include_imports \
    --include_source_info \
    --proto_path=. \
    --descriptor_set_out=api_descriptor.pb \
    --python_out=generated_pb2 \
    --grpc_python_out=generated_pb2 \
    bookstore.proto

2) Deploy the proto descriptor file (api_descriptor.pb) and the configuration file using the gcloud command-line tool:

gcloud endpoints services deploy api_descriptor.pb api_config.yaml

3) Generate gRPC code using Python plugin:

python -m grpc_tools.protoc -I../../protos --python_out=. --grpc_python_out=. ../../protos/helloworld.proto

4) Build the final Docker image to deploy on Google Compute Engine. Resulting Docker image should include:

Generated gRPC code from step 3).
Any additional Python packages required by the gRPC server.

Step 4) builds the 'gRPC Server' (rightmost blue box in the accompanying diagram) using the following Dockerfile:

FROM gcr.io/google_appengine/python:latest

WORKDIR .
EXPOSE 8081
ENTRYPOINT ["python", "server.py"]

ADD requirements.txt .
ADD protos ./protos

RUN mkdir out

RUN apt-get update && \
    apt-get install -y python2.7 python-pip && \
    pip install -r requirements.txt


RUN python \
    -m grpc_tools.protoc \
    --python_out=out \
    --grpc_python_out=out \
    --proto_path=. \
    bookstore.proto

I'm migrating these build steps to Google's Cloud Build.

AFAICT my high-level build steps should map onto Cloud Builder official builder images.

1) ???

2) Use cloud-builders/gcloud/ to run gcloud commands.

3) ???

4) Use cloud-builders/docker to build 'gRPC Server' Docker image.

Steps 2) and 3) already have cloud builders available (see GoogleCloudPlatform/cloud-builders).

However, I'm unsure how to migrate steps 1) and 3) to Cloud Build. Both steps require running a Python plugin which is not available in a base Linux Docker image.

AFAICT step 1) should produce a Cloud Build artifact for api_descriptor.pb and save to a Cloud Storage Bucket.

What Linux Docker image would be fastest / lightest to run the Python gRPC plugin when generating API descriptor files?
Should the aforementioned API descriptor become a Cloud Build artifact and saved to a Cloud Storage Bucket?
- ...in order to use gcloud to deploy the API to Cloud Endpoints.

Jack Jack · Accepted Answer · 2018-11-18T13:43:33

I got this working a few months ago. I don't know if I did it the "right" way. Judge for yourself :p

TL,DR; If you just want to use protoc with Google Cloud Build, I have submitted a protoc builder to the cloud builders community GitHub repository which has been accepted. See cloud-builders-community/protoc.

Detail; My solution relies on creating the protoc Custom Build Step. This creates a Docker container image the Cloud Build worker pulls and runs when it needs to run protoc.

You only need two files to create the Custom Build Step, protoc:

cloudbuild.yaml - tells Google Cloud Builder how to build a Docker image.
Dockerfile - tells Docker how to build the environment containing the protoc binary.

This was literally my local directory structure to achieve step 1:

.
├── cloudbuild.yaml
└── Dockerfile

The Docker file is where the protoc command is installed, and is the more complex of the two files:

FROM ubuntu

ARG PROTOC_VERSION=3.6.1
ARG PROTOC_TARGET=linux-x86_64
ARG ASSET_NAME=protoc-${PROTOC_VERSION}-${PROTOC_TARGET}.zip

RUN apt-get -qy update && apt-get -qy install python wget unzip && rm -rf /var/lib/apt/lists/*

RUN echo "${PROTOC_VERSION}/${ASSET_NAME}"

RUN wget https://github.com/google/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-${PROTOC_TARGET}.zip && \
unzip ${ASSET_NAME} -d protoc && rm ${ASSET_NAME}

ENV PATH=$PATH:/protoc/bin/
ENTRYPOINT ["protoc"]
CMD ["--help]

Breaking this down:

Define the first read only layer of the final "protoc" image we want to end-up with. I chose Ubuntu because it's the what I run locally. Any minimal Linux "base image" will do but it must have the following binaries installed: apt-get, wget, unzip, and rm:

FROM ubuntu

Set-up some variables that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag:

ARG PROTOC_VERSION=3.6.1

ARG PROTOC_TARGET=linux-x86_64

ARG ASSET_NAME=protoc-${PROTOC_VERSION}-${PROTOC_TARGET}.zip

Run apt-get -qy update to "resynchronize the package index files from their sources". q omits progress indicators, y assumes yes as an answer to any prompts encountered:

RUN apt-get -qy update

Install Python, Wget (retrieves content from web server), and Unzip.

RUN apt-get -qy install python wget unzip

Remove any files created as part of the previous steps (and that are no longer needed):

RUN rm -rf /var/lib/apt/lists/*

The previous three RUN instructions can be combined into one:

RUN apt-get -qy update && apt-get -qy install python wget unzip && rm -rf /var/lib/apt/lists/*

Use the ENV instruction to update the PATH environment to include the location of the protoc binary in the final environment (image).

ENV PATH=$PATH:/protoc/bin/

Set the ENTRYPOINT of the image such that the image runs as a protoc executable. Not, since the previous step added protoc to $PATH, we need only specify the binary to run (not the full path):

ENTRYPOINT ["protoc"]

Use the CMD instruction so that if no options are provided when running the protoc image, protoc --help will run:

CMD ["--help]

That's all we need to define an executable protoc Docker image. However, it's not yet a Custom Build Step that can be used in Google's Cloud Build environment. We must define the custom build step using cloudbuild.yaml:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    args:
      [
        'build',
        '--tag',
        'gcr.io/$PROJECT_ID/protoc',
        '--cache-from',
        'gcr.io/$PROJECT_ID/protoc',
        '.',
      ]
images: ['gcr.io/$PROJECT_ID/protoc']

This file will generate an artefact gcr.io/my-cloud-project-id/protoc which can be used to run protoc in Google Cloud Build. Example usage of this custom build step:

steps:
  - name: 'gcr.io/$PROJECT_ID/protoc'
    args:
      [
        '--include_imports',
        '--include_source_info',
        '--proto_path',
        '.',
        '--descriptor_set_out',
        'api_descriptor.pb',
        'v1/my-api-proto.proto',
      ]

Cloud Build automatically replaces $PROJECT_ID with your project ID, so, the name will reference the artefact: gcr.io/my-cloud-project-id/protoc. Since this is an executable Docker image (defined with ENTRYPOINT ["protoc"]), it's equivalent to running locally:

protoc --include_imports --include_source_info --proto_path . --descriptor_set_out api_descriptor.pb v1/my-api-proto.proto

So, in answer to my question, both 1) and 3) can use the protoc custom build step to run in Google Cloud Build.

How to use the gRPC Python Plugin with Docker and Google Cloud Builds?

1 Answers