Error loading multiple files to bigquery too many positional args

Question

~edited: I'm running the bq command line using my VM instance in Google Compute Engine

Ive been trying to load multiple csv files to bigquery using bq command line, and i keep getting this error

Too many positional args, still have ['/home/username/csvschema.json']

All my files contain the same schema since I copied and paste it only and rename for testing purposes. So not sure why I keep getting this error. [testFiles_1.csv, testFiles_2.csv, testFiles_3.csv]

These are the steps I took: 1. Created my bigquery table and manually insert 1 file there so I dont need to manually add schema, but rather auto detect. 2. Then, I type this command:

bq load --skip_leading_rows=1  gcstransfer.testFile /home/username/testfile_*.csv /home/username/csvschema.json

My schema contains by running the bq show --format=prettyjson dataset.table

[
{
"mode": "NULLABLE", 
"name": "Channel", 
"type": "STRING"
}, 
{
"mode": "NULLABLE", 
"name": "Date", 
"type": "INTEGER"
}, 
{
"mode": "NULLABLE", 
"name": "ID", 
"type": "STRING"
}, 
{
"mode": "NULLABLE", 
"name": "Referral", 
"type": "STRING"
}, 
{
"mode": "NULLABLE", 
"name": "Browser", 
"type": "STRING"
}
]

I tried omitting the JSON part, but I get this error instead: BigQuery error in load operation: Error decoding JSON schema from file /home/username/testfile_2.csv: No JSON object could be decoded To specify a one-column schema, use "name:string".

ECris ECris · Accepted Answer · 2020-01-23T15:16:54

Looks like you cannot use wildcards when loading from local data source. For this you can upload the files to a GCS Bucket and load them from there. See the Limitations paragraph in the docs: https://cloud.google.com/bigquery/docs/loading-data-local

Wildcards and comma separated lists are not supported when you load files from a local data source. Files must be loaded individually.

Error loading multiple files to bigquery too many positional args

1 Answers