Loading Avro Data into BigQuery via command-line?

Question

I have created an avro-hive table and loaded data into avro-table from another table using hive insert-overwrite command.I can see the data in avro-hive table but when i try to load this into bigQuery table, It gives an error. Table schema:-

CREATE TABLE `adityadb1.gold_hcth_prfl_datatype_acceptence`(  
  `prfl_id` bigint,     
  `crd_dtl` array< struct < cust_crd_id:bigint,crd_nbr:string,crd_typ_cde:string,crd_typ_cde_desc:string,crdhldr_nm:string,crd_exprn_dte:string,acct_nbr:string,cre_sys_cde:string,cre_sys_cde_desc:string,last_upd_sys_cde:string,last_upd_sys_cde_desc:string,cre_tmst:string,last_upd_tmst:string,str_nbr:int,lng_crd_nbr:string>>)        
STORED AS AVRO;

Error that i am getting:-

Error encountered during job execution:
Error while reading data, error message: The Apache Avro library failed to read data with the follwing error: Cannot resolve:

I am using following command to load the data into bigquery:-

bq load --source_format=AVRO dataset.tableName avro-filePath

I suppose that the "avro-filePath" is not a HDFS path, but either a GCS path or a local (Linux filesystem) path? — Sourygna
@Sourygna,You are right.avro-filePath is GCS PATH.Is it relevant to error? — Vishwanath Sharma
If it's a GCS path it's ok. (I just wanted to be sure it was not HDFS). Don't you have more error message after the words "Cannot resolve:" ? — Sourygna

hlagos hlagos · Accepted Answer · 2018-10-23T15:32:53

Make sure that there is data available in your gs folder where you are pointing and the data contains the schema (it should if your created it from Hive). Here you have an example of how load data

bq --location=US load --source_format=AVRO  --noreplace my_dataset.my_avro_table gs://myfolder/mytablefolder/part-m-00001.avro

Loading Avro Data into BigQuery via command-line?

1 Answers