Google Cloud Build using volumes on Dockerfile

Question

I'm trying to build a build pipeline for my application and share a specific folder between steps using volumes.

The problem is because on my first step (unit-tests) I have to install all the libs on the requirements.txt to be able to run my unit tests. And after that I have to build my application running my Dockerfile in other step. I don't want to re-install all the requirements again, so, I thought in copy the requirements already installed and paste them inside the docker build step. Am I able to do that? I followed this thread and tried to replicate to my reality but I still have problems.

Passing files from Google Cloud Container Builder to Docker build task

Here is a sample of what I've done:

My cloudbuild.yaml:

- id: unit-tests
  name: python:3-alpine
  entrypoint: sh
  dir: my-dir
  args:
    - -c
    - |
      apk add --virtual build_dependencies build-base
      apk add --no-cache libstdc++
      pip install --no-cache-dir -r requirements.txt
      apk del build_dependencies
      python -m unittest discover --verbose
      cp -Rv /usr/local/lib/python3.8/site-packages/* /requirements
  volumes:
    - name: 'requirements'
      path: /requirements

- id: docker-build
  name: gcr.io/cloud-builders/docker
  dir: my-dir
  args:
  - build
  - --tag=gcr.io/${PROJECT_ID}/eta:test
  - '.'
  volumes:
    - name: 'requirements'
      path: /requirements
  waitFor: ['unit-tests']

images:
- 'gcr.io/$PROJECT_ID/eta:test'

timeout: 3600s

And here is my Dockerfile:

from python:3-alpine

RUN set -eux; \
    apk add --virtual build_dependencies build-base; \
    apk add --no-cache libstdc++; \
    apk del build_dependencies; 

COPY /requirements/* /usr/local/lib/python3.8/site-packages/
COPY . /app
RUN ["chmod", "+x", "/app/entry.sh"]
WORKDIR /app
ENTRYPOINT [ "/app/entry.sh" ]

I can't locate the /requirements folder inside my dockerfile, here is the error message:

Step #1 - "docker-build": Step 3/7 : COPY /requirements/* /usr/local/lib/python3.8/site-packages/
Step #1 - "docker-build": COPY failed: no source files were specified
Finished Step #1 - "docker-build"
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

I'm not sure if what I'm doing is the right way.

What you're trying should succeed. You could try adding an intermediate step that will list the contents of /requirements to confirm the prior step succeeds. I'll try to repro. NB You can also add options at the root and define the volume(s) there and they'll be applied across all steps. — DazWilkin

DazWilkin DazWilkin · Accepted Answer · 2020-10-05T18:06:59

The Cloud Build (VM) persists /workspace across steps so you may create e.g. /workspace/requirements and use requirements in subsequent steps.

PROJECT=[[YOUR-PROJECT]]
BILLING=[[YOUR-BILLING]]

gcloud projects create ${PROJECT}

gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

gcloud services enable cloudbuild.googleapis.com \
--project=${PROJECT}

Then:

steps:

  - id: also
    name: busybox
    entrypoint: sh
    dir: x
    args:
      - -c
      - |
        mkdir -p /workspace/requirements
        echo "Hello Freddie" > /workspace/requirements/freddie.txt

  - id: after
    name: gcr.io/cloud-builders/docker
    args:
      - build
      - --tag=gcr.io/${PROJECT_ID}/test
      - .
images:
  - gcr.io/${PROJECT_ID}/test

And Dockerfile:

FROM busybox

WORKDIR /test

COPY /requirements/* .

RUN ls -la

ENTRYPOINT ["more","/test/freddie.txt"]

And:

gcloud builds submit . \
--config=./cloudbuild.yaml \
--project=${PROJECT}

Yields:

BUILD
Starting Step #0 - "also"
Step #0 - "also": Pulling image: busybox
Step #0 - "also": Using default tag: latest
Step #0 - "also": latest: Pulling from library/busybox
Step #0 - "also": Digest: sha256:2ca5e69e
Step #0 - "also": Status: Downloaded newer image for busybox:latest
Step #0 - "also": docker.io/library/busybox:latest
Finished Step #0 - "also"
Starting Step #1 - "after"
Step #1 - "after": Already have image (with digest): gcr.io/cloud-builders/docker
Step #1 - "after": Sending build context to Docker daemon   7.68kB
Step #1 - "after": Step 1/5 : FROM busybox
Step #1 - "after":  ---> 6858809bf669
Step #1 - "after": Step 2/5 : WORKDIR /test
Step #1 - "after":  ---> Running in 7e075adce9eb
Step #1 - "after": Removing intermediate container 7e075adce9eb
Step #1 - "after":  ---> 78f7b2faec16
Step #1 - "after": Step 3/5 : COPY /requirements/* .
Step #1 - "after":  ---> 1ec6a35c1deb
Step #1 - "after": Step 4/5 : RUN ls -la
Step #1 - "after":  ---> Running in 943ddad6434c
Step #1 - "after": total 12
Step #1 - "after": .
Step #1 - "after": ..
Step #1 - "after": freddie.txt
Step #1 - "after": Removing intermediate container 943ddad6434c
Step #1 - "after":  ---> 634072b8aec4
Step #1 - "after": Step 5/5 : ENTRYPOINT ["more","/test/freddie.txt"]
Step #1 - "after":  ---> Running in 13c43b5ca924
Step #1 - "after": Removing intermediate container 13c43b5ca924
Step #1 - "after":  ---> f0ebfe1e8e33
Step #1 - "after": Successfully built f0ebfe1e8e33
Step #1 - "after": Successfully tagged gcr.io/.../test:latest
Finished Step #1 - "after"
PUSH

The problem using volumes is that these won't be in the docker build's context. So, if you'd prefer to use volumes, you'd need to copy the volume's content into /workspace.

Then, when you run the gcr.io/cloud-builders/docker step, /workspace is mounted as the root and so you may access /workspace/requirements as /requirements:

Then:

steps:
  - id: before
    name: busybox
    entrypoint: sh
    dir: x
    args:
      - -c
      - |
        echo "Hello Freddie" > /requirements/freddie.txt
    volumes:
      - name: requirements
        path: /requirements

  - id: during
    name: busybox
    entrypoint: sh
    dir: x
    args:
      - -c
      - |
        more /requirements/freddie.txt
    volumes:
      - name: requirements
        path: /requirements

  - id: fix
    name: busybox
    entrypoint: sh
    args:
      - -c
      - |
        ls -la /workspace
        ls -la /requirements
        mkdir -p /workspace/requirements
        cp -r /requirements /workspace/requirements
        ls -la /workspace/requirements
    volumes:
      - name: requirements
        path: /requirements

  - id: after
    name: gcr.io/cloud-builders/docker
    args:
      - build
      - --tag=gcr.io/${PROJECT_ID}/test
      - .

images:
  - gcr.io/${PROJECT_ID}/test

Yields:

BUILD
Starting Step #0 - "before"
Step #0 - "before": Pulling image: busybox
Step #0 - "before": Using default tag: latest
Step #0 - "before": latest: Pulling from library/busybox
Step #0 - "before": Digest: sha256:2ca5e69e
Step #0 - "before": Status: Downloaded newer image for busybox:latest
Step #0 - "before": docker.io/library/busybox:latest
Finished Step #0 - "before"
Starting Step #1 - "during"
Step #1 - "during": Already have image: busybox
Step #1 - "during": Hello Freddie
Finished Step #1 - "during"
Starting Step #2 - "fix"
Step #2 - "fix": Already have image: busybox
Step #2 - "fix": total 24
Step #2 - "fix": .
Step #2 - "fix": ..
Step #2 - "fix": Dockerfile
Step #2 - "fix": cloudbuild.yaml
Step #2 - "fix": requirements
Finished Step #2 - "fix"
Starting Step #3 - "after"
Step #3 - "after": Already have image (with digest): gcr.io/cloud-builders/docker
Step #3 - "after": Sending build context to Docker daemon  8.192kB
Step #3 - "after": Step 1/5 : FROM busybox
Step #3 - "after":  ---> 6858809bf669
Step #3 - "after": Step 2/5 : WORKDIR /test
Step #3 - "after":  ---> Running in 236aa78f2229
Step #3 - "after": Removing intermediate container 236aa78f2229
Step #3 - "after":  ---> cf05164e8175
Step #3 - "after": Step 3/5 : COPY /requirements/* .
Step #3 - "after":  ---> f5dee710b4d7
Step #3 - "after": Step 4/5 : RUN ls -la
Step #3 - "after":  ---> Running in 8a30309a1536
Step #3 - "after": total 12
Step #3 - "after": .
Step #3 - "after": ..
Step #3 - "after": freddie.txt
Step #3 - "after": Removing intermediate container 8a30309a1536
Step #3 - "after":  ---> 20e2e99c3818
Step #3 - "after": Step 5/5 : ENTRYPOINT ["more","/test/freddie.txt"]
Step #3 - "after":  ---> Running in cc6a57aaa103
Step #3 - "after": Removing intermediate container cc6a57aaa103
Step #3 - "after":  ---> 0a50096e471c
Step #3 - "after": Successfully built 0a50096e471c
Step #3 - "after": Successfully tagged gcr.io/.../test:latest
Finished Step #3 - "after"
PUSH
Pushing gcr.io/.../test
...
DONE

Google Cloud Build using volumes on Dockerfile

1 Answers