GCP Dataflow not able to find pubsub subscription

1
votes

My GCP Dataflow job immediately fails with this error message:

Workflow failed. Causes: 
Subscription 'projects/project/subscriptions/subscription' not found. 
Please supply an existing subscription.

The subscription does exist, I was able to click on it in the dataflow UI. I've been able to run this job previously, and it worked. It's only when I try redeploying it that it gives the error.

EDIT: this is how I am getting the pubsub subscription

pipeline
.apply(PubsubIO.readProtos(...)
.fromSubscription(options.getSubscription()))
1
google-cloud-platformgoogle-cloud-dataflow
Was the subscription automatically created by the IO connector with a topic or did you create it explicitly? It might have been deleted. You can use gcloud cli to verify if the subscription truly exists: cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/list. - 大ドア東
Can you share the piece of code where you get and connect to the subscription? - guillaume blaquiere
@guillaumeblaquiere: sure thing, updated the post with the code that I'm using to read the pubsub subscription. Like I said, this code worked fine a couple weeks ago, but the last few deploys, I've had to redeploy ~20 times until it picked up the subscription. - Ben Konz
The pubsub subscription 100% exists, it isn't a typo. If I redeploy the job enough times, it works, but it takes several hours of redeploying over and over again - Ben Konz
@guillaumeblaquiere if you/anyone else encounters this issue, it's a bug in GCP that google is working to fix. In the meantime, add pubsub viewer permissions to the service account. - Ben Konz

1 Answers

0
votes

Workflow failed. Causes: Subscription ‘<subscription_name>’ not found. Please supply an existing subscription.

From the error message, it seemed like the pubsub subscription topic is not visible to the dataflow pipeline. Examing the permission on the service account shows that it has "pubsub.subscription.consume" permission already. Nothing looked out of order. So, we reported this issue to the Google Cloud team, and it turns out other developers are also facing a similar issue. Google Cloud has created a bug for the same.

Meanwhile, those who are facing this issue in the dataflow deployment can fix it by adding “pubsub.subscriptions.get” permission to the dataflow service account.