Data transfer from Hive to Google Storage/Big Query

0
votes

I have some Hive tables in an on-premise hadoop cluster.
I need to transfer the tables to BigQuery in google cloud.

Can you suggest any google tools or any open source tools for the data transfer?

Thanks in advance

1
apache-sparkgoogle-bigquerygoogle-cloud-platformgoogle-cloud-storagegoogle-cloud-dataflow

1 Answers

1
votes

BigQuery can import Avro files.

This means you can do something like INSERT overwrite table target_avro_hive_table SELECT * FROM source_hive_table;

You can then load the underlying .avro files into BigQuery via the bq command line tool or using the console UI:

bq load --source_format=AVRO your_dataset.something something.avro