DataProc BigQuery Connector Access Across Projects

Question

I am writing a Spark Job to run on a DataProc cluster in Project A but the job itself will pull data from a BigQuery instance in Project B using the BigQuery Connector. I have owner privileges for both project, but the job is run using a service account. The response I'm getting in the stack trace is this:

{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Access Denied: Table ABC:DEF.ghi: The user [email protected] does not have bigquery.tables.get permission for table ABC:DEF.ghi.",
    "reason" : "accessDenied"
  } ],
  "message" : "Access Denied: Table ABC:DEF.ghi: The user [email protected] does not have bigquery.tables.get permission for table ABC:DEF.ghi."
}

Dennis Huo Dennis Huo · Accepted Answer · 2018-03-22T20:11:54

As you noticed, Dataproc clusters run on behalf of service accounts instead of individual users; this is intentional, since different users may be creating Dataproc clusters in a shared project where they do not want their personal permissions to leak across other members of the org using the same project, and instead should define permissions according to service accounts which each represent a particular scope of workloads.

In this case, all you have to do is go into project B and add the service account from project A as one of the roles that can access BQ in project B. If it's not a complex arrangement withlots of users and different teams, you could just add it as "project viewer" on project B, otherwise you'll want something more fine-grained like a "bigquery viewer" or "bigquery editor".

Add that service account the same way you would add any user to project B.

DataProc BigQuery Connector Access Across Projects

1 Answers