Cloud Dataflow job reading from one Bigquery project and writing to another BigQuery project

0
votes

I'm implementing a Cloud Dataflow job on GCP that needs to deal with 2 GCP projects. Both input and output are Bigquery partitionned tables. The issue I'm going through now is that I must read data from a project A and write it into a project B.

I havent seen anything related to cross project service accounts and I can't give Dataflow two different credential key either which is a bit annoying ? I don't know if someone else went through that kind of architecture or how you dealt with it.

2
google-cloud-platformgoogle-bigquerygoogle-cloud-dataflowapache-beamservice-accounts
Could you add more specifics to your question? Is using Dataflow a hard requirement or more a preference/current solution? What are you doing with the 2 BigQuery tables? Reading them into Dataflow's PCols or actually doing something else? From the sound of it, you might be using some templatized Dataflow job, if so, which one? – Francesco Galletta
If the answers do not work for you, please comment here, and I'll try to help you figure this out. I believe Brent's answer should do the trick. – Pablo
So: yes Dataflow is a hard architecture requirement like having 2 BigQuery datasets in separate environment: these are like my architecture constraints. So I read the data from tables in dataset A then apply custom transformations for ML model preparation and then push this to dataset B, I'm not using templatized jobs... – Alex

2 Answers

4
votes

I think you can accomplish this with the following steps:

  1. Create a dedicated service account in the project running the Dataflow job.
  2. Grant the service account the Dataflow Worker and BigQuery Job User roles. The service account might need additional roles based on the full resource needs of the Dataflow job.
  3. In Project A, grant the service account the BigQuery Data Viewer role to either the entire project or to specific datasets.
  4. In Project B, grant the service account the BigQuery Data Editor role to either the entire project or to specific datasets.
  5. When you start the Dataflow job, override the service account pipeline option supplying the new service account.
0
votes

It is very simple. you need to give required permission/access to your service account from both the project.

So you need only service account which has required access/permission in both the project

Hope it helps.