Insert into BigQuery without a well defined struct

Question

I'd like to insert any type of JSON directly into BigQuery, but haven't found any good way of doing this? All ways assume I have a well defined struct that I insert. It seems it was possible with the old deprecated api but not with the new "cloud.google.com/go/bigquery" package.

I'd like to have /api/table_name/insert be able to take any type of json and insert it into BigQuery as both the client and server knows the schema the endpoint should just forward it.

Thanks

BigQuery is not a "no-sql" database, it requires a schema. So "I'd like to insert any type of JSON directly into BigQuery" won't work without having to define a schema first. But then, if you already have a schema, you can have a matching struct to it. — icza
I have a schema defined, and a method do dynamically create more schemas as needed, but I dont want to recompile the go code with new structs every time I add a new schema — Nixarn

Itamar Lavender Itamar Lavender · Accepted Answer · 2019-06-15T20:48:00

seems like this was replaced with ValueSaver which you can implement like:

type genericRecord map[string]bigquery.Value

func (rec genericRecord) Save() (map[string]bigquery.Value, string, error) {
    insertID := uuid.New().String()
    return rec, insertID, nil
}

var data []*genericRecord
json.Unmarshal(<YOUR JSON BYTE>, &data)

ctx := context.Background()
client, err := bigquery.NewClient(ctx, <YOUR PROJECT ID>)
ins := client.Dataset(<DATASET>).Table(<TABLE>).Inserter()
ins.Put(ctx, data)

Insert into BigQuery without a well defined struct

1 Answers