Pipelining in GCP

Question

I have lots of data being collected regularly from data sources and stored on GCS buckets. Currently they are stored in one local region (London). We are pipelining these data with ETL using combination of GCP services (i.e. composer, dataflow or cloud functions etc.) to BigQuery tables that reside in the same region (e.g. europe-west2 London).

in addition to this, I would like to pipeline these data (e.g. buckets) to different BQ tables located in different region.

So, in summary the output is big query tables located in two different regions, as follows:

GCS buckets: project 1, buckets in London (europe-west2)

Output1: BQ table on project1 region1(europe-west2 London) - (existing)

expected: output2: BQ table on project2 region2 (EU) - either multiregion or alternatively Belgium

Can i do this and how should I do this efficiently?

Appreciate suggestion and help on this. Thank you!

Answer 1

One option is using BigQuery's cross region copy of your dataset. The feature is currently in Beta and cost free for now (please note it may not always be free).

Although the service is in limited location, europe-west2 and EU are both supported.

Answer 2

Apart from the cross region copy , you can design the pipeline where you the read files from GCS and pipe it to Pub-Sub Topic. Then from Pub-Sub topic you can send it region specific BigQuery ?