Very slow data loading in BigQuery

2
votes

I'm trying to bulk load some records to BigQuery, but it takes a long time to upload even a few thousands records.

I'm using the following command to load a gzipped JSON file. The file has ~2k rows with ~200 columns each:

./bin/bq load --project_id=my-project-id --source_format=NEWLINE_DELIMITED_JSON dataset.table /tmp/file.json.gz
Waiting on bqjob_r3a269dd7388c7b8e_000001579a6e064f_1 ... (50s)
Current status: DONE

This command takes ~50 seconds to load the records. As I want to load at least 1 million records, this would take ~7 hours, which seems too much for a tool that is supposed to handle petabytes of data.

Is it possible to speed up the process?

1
google-bigquery
Time is not progressive it won't take that much even for 1 million too. Run it. - Pentium10

1 Answers

1
votes

Try using --nosync flag. This will start an Asynchronous job over bigQuery, found this having much better performance.

Optimally, I would suggest to store file.json.gz inside Google Cloud Storage. 

./bin/bq load --nosync