Error in bq load “Could not convert value to string”

Question

I tried to load logs from Google Cloud Storage to BigQuery by the bq command and I've got this error "Could not convert value to string".

my example data

{"ids":"1234,5678"}
{"ids":1234}

my example schema

[
    { "name":"ids", "type":"string" }
]

It seems IDs can't convert by none quote at single ID. Data is made with fluent-plugin-s3, but more than one ID connected by a comma can be bound up with a quotation and isn't made single id.

How can I load these data to BigQuery?

Thanks in advance

Answer 1

Well check different fluentd plugins that can help you, maybe

• https://github.com/lob/fluent-plugin-json-transform
• https://github.com/tarom/fluent-plugin-typecast