Cloud dataflow tutorial get Error "Failed to create a workflow job: Dataflow API has not been..."

Question

I am trying to get through the cloud DataFlow tutorial. I can run it locally and previously I was able to execute it remotely, but today I started getting the following error.

I am running

 mvn compile exec:java \
      -Dexec.mainClass=com.example.WordCount \
      -Dexec.args="--project=list-cleaner\
      --stagingLocation=gs://disismybucket/staging/ \
      --output=gs://disismybucket/output \
      --runner=DataflowRunner"

And getting the following --

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Dataflow API has not been used in project 563584335869 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/dataflow.googleapis.com/overview?project=563584335869 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
    "reason" : "accessNotConfigured",
    "extendedHelp" : "https://console.developers.google.com"
  } ],
  "message" : "Dataflow API has not been used in project 563584335869 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/dataflow.googleapis.com/overview?project=563584335869 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
  "status" : "PERMISSION_DENIED"
}

It's obviously an auth issue but I have properly set my environment variables for GOOGLE_APPLICATION_CREDENTIALS and I have made sure that my project name is correct. I have disabled and re-enabled the api for this project to no avail. Oddly the project id referenced is not one of my projects but is a project number that seems to show up in other support issues. I have also tried it with the python SDK as well, but no luck with that either. Any help is appreciated.

From the error message, it looks like you did not enable Dataflow API for your project. Hmm, I see you mentioned this is not your recognized "working" project. You might want to get a new credential json from your "working project" and point your GOOGLE_APPLICATION_CREDENTIALS to the new json file. — greeness
Thanks for the response. I have tried getting new credentials as you have described. I also tried created a completely new project from scratch and use those credentials and I have the same issues :-/ — Ryan Peterson
Also make sure your project contains a service account that has the role "Cloud Dataflow service agent" (you can view all your members in IAM). This member has a name like %service_number%@dataflow-service-producer-prod.iam.gserviceaccount.com". — greeness
I have seen this error message before and it's indeed always related to setting GOOGLE_APPLICATION_CREDENTIALS to the correct json file (by the way, this in turn points to another issue, that is, the error message is misleading). How are you creating the credentials json? What permissions does the service account whose key is in the credentials file have? — Lefteris S
Also, you might want to double-check your active configuration by running gcloud config list. You can make sure that you are running from the correct project by using the following commands: gcloud config configurations create $CONFIG, gcloud config set project $PROJECT, gcloud config set account $ACCOUNT — Lefteris S

Ryan Peterson Ryan Peterson · Accepted Answer · 2018-06-02T21:28:53

So as I feared it was something really simple and I am an idiot. Your comments made me look harder at what I was doing around setting the GOOGLE_APPLICATION_CREDENTIALS. I was setting it GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json", but I was not exporting it into the global scope ie export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json" I will say that it is odd behavior that instead of throwing an error about the missing credentials it defaults to some random account

Cloud dataflow tutorial get Error "Failed to create a workflow job: Dataflow API has not been..."

1 Answers