Can I run a setup script when submitting a Dataflow job before running the pipeline?

Question

I'm employing Apache Beam Python SDK 2.0.0 to define a pipeline and I'm trying to run it on the Google Cloud Dataflow.

The code uses the gcs-oauth2-boto-plugin for authentication purposes, but it requires installing some packages[1] via apt-get . So I need to run this code sudo apt-get install gcc python-dev python-setuptools libffi-dev libssl-dev before executing the pipeline.

How can I tell the Dataflow to execute this command in the VM before executing the job? I Know that we can specify a startup script for a Compute Engine, but in Dataflow this is created under the hood, can I do something analogous when submitting a job in Dataflow?

[1] https://cloud.google.com/storage/docs/xml-api/gspythonlibrary

Answer 1

For Python Dependencies

You can pass a --requirements_file argument when you submit your pipeline. This file should contain a line-separated list of package==version, just like the output of the pip freeze command.

For non-Python dependencies

In this case, you will need to pass a --setup_file argument to your pipeline. In the setup.py file, you add a call to setuptools.setup, in which you pass a cmdclass argument with classes that implement your installation logic. The Apache Beam repository contains an example of this.

For more on this, check Managing Pipeline Dependencies.