Dataflow SDK 2.7.0 while attaching a setup 2.10.0

1
votes

I am trying to run a dataflow job, have the following setup.py file:

import setuptools

setuptools.setup(
              name='Some-Name',
              version='0.0.1',
              install_requires=[
                  'apache-beam==2.10.0',
                  'tensorflow-transform==0.12.0'
                  ],
              packages=setuptools.find_packages(),
              )

but when I submit the job to GCP, the SDK shows as 2.7.0 and the job won't run. It will stop after 1 hour with the following error:

Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h.

1
pythongoogle-cloud-platformsdkgoogle-cloud-dataflowapache-beam

1 Answers

1
votes

I fixed this issue by updating the tensorflow-transform and apache-beam in my local virtual environment where I was submitting the dataflow job from. Once I updated my local apache-beam and tensorflow-transform and tensorflow, it seems to work. I'm not sure why since the setup.py stated the versions of the packages.