Data Transfer between two BigQuery Projects using Google Cloud Functions

Question

I have created two projects on Google Cloud Platform say project1 and project2. Project1 has a bigquery dataset named dataset1 which contains a table named table1 which has some contents. Project2 has a bigquery dataset named dataset2 which contains a table named table2 which is empty. I need a python code that will copy/import the table1 and export/copy it to table2 which was initially empty using Google Cloud Functions tool.

Answer 1

1. Understand how to use Python to send a query to BigQuery following the documentation.

2. The query to "copy/import the table1 and export/copy it to table2" you will need is (assuming table2 has exactly same schema as table1):

INSERT INTO project2.dataset2.table2 
SELECT * FROM project1.dataset1.table1;

Answer 2

Find Python code to copy a table here:

• https://cloud.google.com/bigquery/docs/managing-tables#copy-table

The code is:

# from google.cloud import bigquery
# client = bigquery.Client()

source_dataset = client.dataset("samples", project="bigquery-public-data")
source_table_ref = source_dataset.table("shakespeare")

# dataset_id = 'my_dataset'
dest_table_ref = client.dataset(dataset_id).table("destination_table")

job = client.copy_table(
    source_table_ref,
    dest_table_ref,
    # Location must match that of the source and destination tables.
    location="US",
)  # API request

job.result()  # Waits for job to complete.

assert job.state == "DONE"
dest_table = client.get_table(dest_table_ref)  # API request
assert dest_table.num_rows > 0

There's another answer to that question that shows you can do it with INSERT INTO *, but that operation will have the cost of a full table scan - vs free with this one.

(I normally use CREATE TABLE or INSERT INTO because they are more convenient tho)