Google Cloud Build gsutil with python entrypoint

Question

I want to use a python script to download several files from Google Storage by calling gsutil via subprocess.check_call() inside the Python script during a Cloud Build step.

Therefore, I use the following step inside the cloudbuild.yaml:

- name: 'gcr.io/cloud-builders/gsutil'
  entrypoint: 'bash'
  args: ['-c',
         'pip install google-cloud-storage && python /path_to_script/script.py --arg1 $_ARG1 --arg2 $TAG_NAME --arg3 $BRANCH_NAME --arg4 $_ARG4'
  ]

But Cloud Build throws a strange SyntaxError: invalid syntax Python error. I can run the same Python script with the same arguments locally as well as by using python:3.7-sim inside Cloud Build with that SyntaxError:

- name: 'python:3.7-slim'
  entrypoint: /bin/sh
  args: ['-c',
         'pip install google-cloud-storage && python python /path_to_script/script.py --arg1 $_ARG1 --arg2 $TAG_NAME --arg3 $BRANCH_NAME --arg4 $_ARG4'
  ]

The later example is working until gsutil is called, because it is not part of the python:3.7-slim container (therefor expected behavior).

Any helping comments on why gcr.io/cloud-builders/gsutil is throwing that error? Is the entrypoint wrongly specified?

The script pulls a list of all files in the bucket and filters them according to some user input (using substitution variables in cloud build). Then, the files are downloaded to the current workspace in cloud build. Interestingly, the scripts breaks right at the beginning within the function definition. As said before, the scripts itself runs locally as well as within the python:3.7-slim container. — winwin
I can't reproduce your error. That's why I would like to understand the special thing that you do to fail. If you have a minimal code to reproduce the issue, it's perfect! — guillaume blaquiere

winwin winwin · Accepted Answer · 2020-10-19T11:21:39

I found an easier solution by invoking the dedicated gsutil image in combination with wildcards to accomplish a similar filtering as in the python script. See simplified example below:

- name: 'gcr.io/cloud-builders/gsutil'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    gsutil -m cp -r 'gs://bucket/*/' '/workspace'

This actually does the job for me pretty well.

Google Cloud Build gsutil with python entrypoint

1 Answers