How to fix 'Not found: Files /bigstore/project/testing/filename.json' error when loading into Bigquery

1
votes

I'm trying to load multiple json (4000) files into a table in Bigquery using the following command bq load --source_format=NEWLINE_DELIMITED_JSON --replace=true kx-test.store_requests gs://kx-gam-test/store/requests/*, and I am getting the following error:

Error encountered during job execution:
Not found: Files /bigstore/kx-gam-test/store/requests/7fb27d63-5581-43a1-821d-fcf47b3412fd.json.gz
Failure details:
 - Not found: Files /bigstore/kx-gam-test/store/requests/93b54246-2284-4b85-8620-76657f4a338b.json.gz
 - Not found: Files /bigstore/kx-gam-test/store/requests/fd24a53d-2c49-4f66-bf54-a7ccf14a1cfe.json.gz
 - Not found: Files /bigstore/kx-gam-test/store/requests/35a27032-930c-456a-846d-67481a21e52d.json.gz

I am not sure why it is not working, is it possibly due to the number of files I am trying to load? And what is this bigstore folder prefixed in front of my GCS bucket?

I would like to highlight that the folder structure is such that there are some folders inside of kx-gam-test/store/requests, and I would want to load the json gzip files inside all these folders.

2
google-bigquery

2 Answers

0
votes

According to the documentation:

BigQuery does not support source URIs that include multiple consecutive slashes after the initial double slash.

Also, here is some additional info to consider when loading data to cloud storage.

0
votes

Few things you can check:

  1. Make sure that you have the necessary permissions

  2. Make sure that the files do exist in GCS

  3. Do you have any process which deletes the file after the loading? Check the audit logs for any traces whether the file might have been deleted while BQ is actually reading/loading it.