Cloudbuild with Python 2.7 and Apache Beam

Question

I created a pipeline with Apache Beam on Python 2.7 that runs on Google Dataflow. This pipeline works well when I deploy it locally from my laptop. I now wish to deploy it via CloudBuild. This is my cloudbuild.yaml file:

steps:
  - name: "docker.io/library/python:2.7"
    args: ["pip", "install", "-t", "/workspace/lib", "-r", "requirements.txt"]
  - name: "docker.io/library/python:2.7"
    args: ["python2", "tests.py"]
    env: ["PYTHONPATH=/workspace/lib"]

When the CloudBuild is triggered, it successfully installs all the requirements, but then when it tries to import apache_beam in the tests.py file, I recieve the following error:

   File "tests.py", line 3, in <module>
     import apache_beam as beam
   File "/workspace/lib/apache_beam/__init__.py", line 97, in <module>
     from apache_beam import coders
   File "/workspace/lib/apache_beam/coders/__init__.py", line 19, in <module>
     from apache_beam.coders.coders import *
   File "/workspace/lib/apache_beam/coders/coders.py", line 29, in <module>
     import google.protobuf.wrappers_pb2
 ImportError: No module named google.protobuf.wrappers_pb2

In the requirements.txt file I have inter alia the following:

apache-beam==2.16.0
protobuf==3.11.0

Note: All the necessary requirements are listed in requirements.txt, since I can deploy the pipeline locally.

Hi, did you check my answer? Are you still having the same problems? — Juancki
Thank you for taking some time to review the thread! Happy coding! — Juancki

Juancki Juancki · Accepted Answer · 2020-01-28T10:31:00

I was able to reproduce your issue with only another file named tests.py with one line:

import apache_beam as beam

It seems that it is a known issue in some communities.

First of all, this error does not happen if instead of using the deprecated Python2 you use instead Python3.7. I suppose though that Python2.7 is required.

I created a Dockerfile such as:

FROM python:2

WORKDIR /usr/src/app

COPY requirements.txt ./

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "./tests.py" ]

And then, changed the cloudbuild.yaml:

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/quickstart-image', '.' ]

which builds succesfully:

Step 5/6 : COPY . .
 ---> 082407f9672d
Step 6/6 : CMD [ "python", "./tests.py" ]
 ---> Running in 7d53e96370f9
Removing intermediate container 7d53e96370f9
 ---> d60a6d473d21
Successfully built d60a6d473d21

Cloudbuild with Python 2.7 and Apache Beam

1 Answers