Schedule task to load BigQuery table into Apache Ignite

1
votes

I have a use case where we need to periodically load BigQuery table in to a cache and support SQL query from there. I'm doing researching on Apache Ignite and think it could be a good fit to our use case. Only that it's not clear to me yet how I can get auto-load from BigQuery. By "auto-load" I mean to keep Apache Ignite updated with BigQuery table data and let this updating transparent to applications. In most cases, our BigQuery tables are updated by other scheduled jobs/queries with intervals from 5 minutes to 1 month.

I'm new to Ignite, and I guess my questions are as the following:

  1. Is this a feature supported in Ignite already? (I couldn't find any)

  2. Or is there any exiting pluggins already? (I couldn't find any)

  3. how to implement the auto-load cache for BigQuery using Ignite?

2
google-bigqueryignite

2 Answers

0
votes

You can do this once with Cache Store / loadCache(), but doing this every few minutes is infeasible. You may wish to design a BigQuery streamer to Apache Ignite, if it supports pushing of deltas.

0
votes

If Google BigQuery doesn't open its changelog files for CDC tools then find how to capture those updates differently and stream them to Ignite via its IgniteDataStreamer API. There should be a way to capture the changes with some pub/sub mechanism.