Load JSON data into BigQuery table

Question

I'm trying to load simple JSON data into BigQuery table the following way:

$ bq load \
    --apilog \
    --source_format=NEWLINE_DELIMITED_JSON \
    my_dataset.my_table \
    ./input.json ./schema.json

but get the following error message:

Upload complete.
Waiting on bqjob_xxxx_xxx ... (3s) Current status: DONE
BigQuery error in load operation: Error processing job 'my_project_id:bqjob_xxxx_xxx': CSV table encountered too many errors, giving up. Rows: 1; errors: 1.
Failure details:
- file-00000000: Error detected while parsing row starting at
position: 0. Error: Data between close double quote (") and field
separator.

It complains about some CSV error, but I'm trying to load JSON (--source_format=NEWLINE_DELIMITED_JSON)

My input.json contains this data:

{"domain":"stackoverflow.com","key":"hello","value":"world"}

My schema.json is the following:

[
    {
        "name": "domain",
        "type": "string",
        "mode": "nullable"
    },
    {
        "name": "key",
        "type": "string",
        "mode": "nullable"
    },
    {
        "name": "value",
        "type": "string",
        "mode": "nullable"
    }
]

bq version 2.0.25:

$ gcloud version | grep ^bq
bq 2.0.25

Answer 1

The problem here is that the flag apilog expects a string as input. This command should work for you:

bq load \
    --apilog '' \
    --source_format=NEWLINE_DELIMITED_JSON \
    my_dataset.my_table \
    ./input.json ./schema.json

Empty string sends output to stdout. If you want to save the log to a local file then you can just send a non-empty string, such as --apilog 'localfile_name'.

Answer 2

BQ command says:

USAGE: bq.py [--global_flags] <command> [--command_flags] [args]

As you see there are global_flags and command_flags

For the global_flags that have values you need to use the equal sign:

--flag=value

The command_flags are either boolean:

--[no]replace

Or they take arguments that must follow the flag:

--source_format NEWLINE_DELIMITED_JSON

Also do not mix global and command flags: apilog is a global flag. I would rewrite your command into:

$ bq --apilog load \
    --source_format NEWLINE_DELIMITED_JSON \
    my_dataset.my_table \
    ./input.json ./schema.json