How to fix "...problem refreshing your credentials" error when the dataflow API and service accounts enabled and present

1
votes

Every time I try to run a job in dataflow, I get the following error:

"Workflow failed. Causes: There was a problem refreshing your credentials. Please check: 1. Dataflow API is enabled for your project. 2. There is a robot service account for your project: service-[project number]@dataflow-service-producer-prod.iam.gserviceaccount.com should have access to your project. If this account does not appear in the permissions tab for your project, contact Dataflow support., There is no cloudservices robot account for your project. Please ensure that the Dataflow API is enabled for your project."

I have disabled and reenabled the dataflow API. I have also tried disabling and reenabling all of the other dependent APIs as well.

I checked the IAM page and the service account is there and has the role "Cloud Dataflow Service Agent" I even tried giving the service account permission of "Owner" for the project to see if that would help and it didn't.

Any other ideas? Thanks in advance for your help!

1
google-cloud-dataflow

1 Answers

1
votes

This drove me crazy for a long time. Finally figured out that I needed to add the service account and associated credentials to my pipeline options. A bit like this:

final static List<String> SCOPES = Arrays.asList(
            "https://www.googleapis.com/auth/cloud-platform",
            "https://www.googleapis.com/auth/devstorage.full_control",
            "https://www.googleapis.com/auth/userinfo.email",
            "https://www.googleapis.com/auth/datastore",
            "https://www.googleapis.com/auth/pubsub");
GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream("/path/to/service/account/keyfile.json"));
options.setServiceAccount("[email protected]");
options.setGcpCredential(credentials.createScoped(SCOPES));

Hope this helps other people who have had the same issue