Size of data block in avro is larger than the maximum allowed value 16777216

Question

I am trying to load Avro data into Bigquery.So i converted ORC data into AVRO by running INSERT OVERWRITE COMMAND in hive.When i try to load data in Bigquery using bq command line tool, I am getting this error:-

"message": "Error while reading data, error message: Avro parsing error in position 397707. Size of data block 17378680 is larger than the maximum allowed value 16777216."

Is there any way i can increase this data block size.I couldn't find anything related to this.

Below is the command that i am trying to use to load data.

bq load --source_format=AVRO dataset.table gs://********/gold/offers/hive/gold_hctc_ofr_txt/ingestion_time=20180305/000000_0

Answer 1

Seems like you are actually getting bumped due to BigQuery’s block size limit as defined in this document. You can check on the Row and cell size limits section where it is mentioned that Avro’s block size is 16MB.