Is it possible to load Avro files with Snappy compression into BigQuery?

1
votes

I know that BigQuery supports Avro file upload and I'm successful in loading Avro file into BigQuery.

Using below command,

java -jar avro-tools-1.7.7.jar fromjson --codec snappy --schema-file SourceSchema.avsc Source.json > Output.snappy.avro

I have generated an Avro file with Snappy compression and trying to load into BigQuery but Load job fails with below errors,

Errors:
file-00000000: The Apache Avro library failed to parse file file-00000000. (error code: invalid)

Is it possible to load Avro files with Snappy compression into BigQuery?

2
google-bigqueryavrosnappy
Please make codec as none, as snappy is not supported.This will bloat file size though. – akshat thakar

2 Answers

3
votes

BigQuery only supports DEFLATE and Snappy algorithm for Avro data blocks compression, from the docs (https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro)

Compressed Avro files are not supported, but compressed data blocks are. BigQuery supports the DEFLATE and Snappy codecs.

0
votes

Now BigQuery supports Snappy. See: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro

Compressed Avro files are not supported, but compressed data blocks are. BigQuery supports the DEFLATE and Snappy codecs.