Google BigQuery - Directly upload data from Google Cloud Storage

0
votes

Is there any way to upload data to the Google BigQuery from Google Cloud storage without downloading it by .NET code? Like it could be done on UI.

Here is the sample from the documentation: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage#bigquery-import-gcs-file-csharp

    StorageClient gcsClient = StorageClient.Create();

    using (var stream = new MemoryStream())
    {
        // Set Cloud Storage Bucket name. This uses a bucket named the same as the project.
        string bucket = projectId;
        // If folder is passed in, add it to Cloud Storage File Path using "/" character
        string filePath = string.IsNullOrEmpty(folder) ? fileName : folder + "/" + fileName;
        // Download Google Cloud Storage object into stream
        gcsClient.DownloadObject(projectId, filePath, stream);

        // This example uploads data to an existing table. If the upload will create a new table
        // or if the schema in the JSON isn't identical to the schema in the table,
        // create a schema to pass into the call instead of passing in a null value.
        BigQueryJob job = client.UploadJson(datasetId, tableId, null, stream);
        // Use the job to find out when the data has finished being inserted into the table,
        // report errors etc.

        // Wait for the job to complete.
        job.PollUntilCompleted();
    }
1
c#.netgoogle-bigquerygoogle-cloud-storage

1 Answers

1
votes

Here the way

public void ImportDataFromCloudStorage(BigQueryClient client, string uri, TableReference reference, WriteDisposition disposition = WriteDisposition.WriteIfEmpty)
    {
      BigQueryJob job = client.CreateLoadJob(uri, reference, null, new CreateLoadJobOptions()
      {
        FieldDelimiter = ";",
        SkipLeadingRows = 1,
        AllowQuotedNewlines = true,
        WriteDisposition = disposition
      });

      job.PollUntilCompleted();
    }