Having skimmed the Google Cloud Dataflow documentation, my impression is that worker VMs run a specific predefined Python 2.7 environment without any option to change that. Is it possible to provide a custom VM image for the workers (built with libraries, external commands that the particular application needs). Is it possible to run Python 3 on Gcloud Dataflow?
5
votes
4 Answers
4
votes
Python 3 support in to Apache Beam status: https://beam.apache.org/roadmap/python-sdk/#python-3-support
4
votes
2021 Update
As of today, the answer to both of this questions is YES.
- Python 3 is supported on Dataflow.
- Custom container images are supported on Dataflow, see this SO answer, and this docs page.
Is it possible to provide a custom VM image for the workers (built with libraries, external commands that the particular application needs). Is it possible to run Python 3 on Gcloud Dataflow?
No and no to both questions. You're able to configure Compute Engine instance machine type and disk size for a Dataflow job, but you're not able to configure things like installed applications. Currently, Apache Beam does not support Python 3.x.
References:
3
votes
You cannot provide a custom VM image for the workers, but you can provide a setup.py file to run custom commands and install libraries.
You can find more info about the setup.py file here: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/
1
votes
