How to load multiple AVRO files using a bq load command

Question

I'm trying to load multiple AVRO files into big query following this docs:

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro

According the docs the command for do this is :

bq --location=US load --source_format=AVRO [DATASET].[TABLE_NAME] "gs://mybucket/00/*.avro","gs://mybucket/01/*.avro"

I create a script for search files and mount the command like this:

bq load --source_format=AVRO --noreplace foo.bar$123456  "gs://mybucket/foo/36.avro", "gs://mybucket/foo_bar/01.avro", "gs://mybucket/bar/211.avro"

But this only work when i have one file like this:

bq load --source_format=AVRO --noreplace foo.bar$123456 "gs://mybucket/foo/36.avro"

When i try to use the command for multiple file the error is:

Too many positional args, still have ["gs://mybucket/foo_bar/01.avro"]

Its my script to create the commands:

def create_command_bq_load(buckets):
    for x, bucket in enumerate(buckets):
        command =  'bq load --source_format=AVRO --noreplace %s.%s_%s$%s' % (datasetname,  bucket['product'], bucket['event'],  bucket['data_partition'])
        if bucket['files']:
            command_file = ''
            for x in range(len(bucket['files'])):    
                command_file = '%s "%s",' % (command_file, bucket['files'][x])   
                command_file = command_file
            commands.append((command + ' ' + command_file)[:-1])
    return commands

Some help?

Solved, my error is the ' ' character betwen two files... the correct way is this: bq load --source_format=AVRO --noreplace foo.bar$123456 "gs://mybucket/foo/36.avro","gs://mybucket/foo_bar/01.avro","gs://mybucket/bar/211.avro" — Gabriel Velho Ghellere
Hi Gabriel, if that solved your problem, please post it as an answer, thanks. — Pol Ortiz

Gabriel Velho Ghellere Gabriel Velho Ghellere · Accepted Answer · 2019-04-25T14:28:04

Solved, my error is the space ' ' character betwen two files... the correct way is this:

bq load --source_format=AVRO --noreplace foo.bar$123456 "gs://mybucket/foo/36.avro","gs://mybucket/foo_bar/01.avro","gs://mybucket/bar/211.avro"

How to load multiple AVRO files using a bq load command

1 Answers