BigQuery transfer service: which file causes error?

0
votes

I'm trying to load around 1000 files from Google Cloud Storage into BigQuery using the BigQuery transfer service, but it appears I have an error in one of my files:

Job bqts_601e696e-0000-2ef0-812d-f403043921ec (table streams) failed with error INVALID_ARGUMENT: Error while reading data, error message: CSV table references column position 19, but line starting at position:206 contains only 19 columns.; JobID: 931777629779:bqts_601e696e-0000-2ef0-812d-f403043921ec

How can I find which file is causing this error? I feel like this is in the docs somewhere, but I can't seem to find it.

Thanks!

1
google-cloud-platformgoogle-bigquery

1 Answers

2
votes

You can use bq show --format=prettyjson -j job_id_here and will show a verbose error about the failed job. You can see more info about the usage of the command in BigQuery managing jobs docs.

I tried this with a failed job of mine wherein I'm loading csv files from a Google Coud Storage bucket in my project.

Command used:

bq show --format=prettyjson -j bqts_xxxx-xxxx-xxxx-xxxx

Here is a snippet of the output. Output is in JSON format: enter image description here