Google Cloud bigQuery connect ruby API query to table

Question

I have a software product with a Ruby API that generates a table-like output when queried, and I would like to dynamically connect the output to Google Cloud bigQuery.

Having read the documentation, there is a dynamic connector for Google Sheets, and static ETL connectors to PostgreSQL and other (https://cloud.google.com/blog/big-data/2016/05/bigquery-integrates-with-google-drive).

If I have a ruby query that looks like the one below:

ruby productX-api/ruby/query_table.rb param1 param2

and this produces a table from the query:

field1,field2,field3
foo,bar,bar
xyz,abc,def

What options do I have to connect this to bigQuery?

Hmm. I'm a little confused (maybe because I don't know Ruby): "generates a table-like output when queried": what does this mean? Does it return a big string/file e.g. csv? — Graham Polley
@GrahamPolley yes it's a ruby script that returns a big string/file e.g. in csv format. How can I connect this to bigQuery? — 719016
What do you mean by "connect"? You want to load that data into BigQuery, right? — Graham Polley
Either have a daemon that loads it every hour or have a way of having the query connected to the bigQuery table, like it's currently available with Google Sheets. — 719016
Does the result of the ruby call truncate an existing table or is it a new table with each load? — Graham Polley

LundinCast LundinCast · Accepted Answer · 2018-03-16T10:15:53

There's no built-in connector as you'd like but you can pretty easily load the resulting csv file programmatically by using the Google Cloud client library for Ruby. For example:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

file = File.open "my_data.csv"
load_job = table.load_job file

More information here for the particular load_job method.

Google Cloud bigQuery connect ruby API query to table

1 Answers