Google BigQuery data transfer - dataset copy

Question

i have a bigquery dataset in my google project project-everest:evr_dataset, i want to copy the its table data to my another bigquery dataset which is sitting in an another project, project-alps:alp_dataset.

I attempted to use DTS - data transfer service to schedule the ingest job on daily, but i dont see any option to choose the destination dataset from another project? Can enlighten me how enable inter-project DTS?

Amandeep Singh Amandeep Singh · Accepted Answer · 2021-09-28T07:09:55

You can use the below python function to create big query data transfer client and copy datsets from one project to another by specifying source and target project id's. You can also schedule the data transfer. In the method below it is set to 24 hours(daily).

def copy_dataset(override_values={}):
    # [START bigquerydatatransfer_copy_dataset]
    from google.cloud import bigquery_datatransfer

    transfer_client = bigquery_datatransfer.DataTransferServiceClient()

    destination_project_id = "my-destination-project"
    destination_dataset_id = "my_destination_dataset"
    source_project_id = "my-source-project"
    source_dataset_id = "my_source_dataset"
    # [END bigquerydatatransfer_copy_dataset]
    # To facilitate testing, we replace values with alternatives
    # provided by the testing harness.
    destination_project_id = override_values.get(
        "destination_project_id", destination_project_id
    )
    destination_dataset_id = override_values.get(
        "destination_dataset_id", destination_dataset_id
    )
    source_project_id = override_values.get("source_project_id", source_project_id)
    source_dataset_id = override_values.get("source_dataset_id", source_dataset_id)
    # [START bigquerydatatransfer_copy_dataset]
    transfer_config = bigquery_datatransfer.TransferConfig(
        destination_dataset_id=destination_dataset_id,
        display_name="Your Dataset Copy Name",
        data_source_id="cross_region_copy",
        params={
            "source_project_id": source_project_id,
            "source_dataset_id": source_dataset_id,
        },
        schedule="every 24 hours",
    )
    transfer_config = transfer_client.create_transfer_config(
        parent=transfer_client.common_project_path(destination_project_id),
        transfer_config=transfer_config,
    )
    print(f"Created transfer config: {transfer_config.name}")
    # [END bigquerydatatransfer_copy_dataset]
    return transfer_config

Google BigQuery data transfer - dataset copy

1 Answers