how to create table in Google BigQuery from continuously uploaded Kafka text files into Google cloud storage

0
votes

i want to create bigquery table from the cloud storage. Kafka steam uploaded as text files into Cloud storage by every 5 minutes. I want to create bigquery table using that is updating every 5 minutes from the updated files into Bigquery. What is the best way to do this? Please give me some suggestions

1
apache-kafkagoogle-bigquerygoogle-cloud-storage
I would recommend to start with this article. Can you get any benefit from there in the current research? – Nick_Kh
@mk_sta Yes. it's really useful. thanks for sharing. – kevin

1 Answers

2
votes

You could use to detect when a file is uploaded, then execute some code to index that file.

Alternatively, I believe there already exists a BigQuery Kafka Connector, so you could skip GCS unless you need the raw data. (Note: binary files would be cheaper to store than plaintext, and BigQuery supports reading various formats)