I am using the Spark BigQuery connector to query tables and views from a Dataproc cluster, what I saw is that when requesting a view the cache is not used, the connector creates a new temporary table for each view read:
df = spark.read.format('bigquery').option('table', view_name).option('viewsEnabled', 'true').load()
it's not the case when I read from a table, the cache here is used:
df = spark.read.format('bigquery').option('table', table).load()
Thank you
If the connector creates a new temporary table, it means that you don't have cached results for that table because you didn't run any query on it (because it is new and it is created at that moment), or it's a use case that may belong to point 3 below.
However, tables already created or external table queried before and outside of the below list of exceptions are fine
Exceptions to query caching
Query results are not cached:
- When a destination table is specified in the job configuration, the Cloud Console, the bq command-line tool, or the API
- If any of the referenced tables or logical views have changed since the results were previously cached
- When any of the tables referenced by the query have recently received streaming inserts (a streaming buffer is attached to the table) even if no new rows have arrived
- If the query uses non-deterministic functions; for example, date and time functions such as CURRENT_TIMESTAMP() and NOW(), and other functions such as CURRENT_USER() return different values depending on when a query is executed
- If you are querying multiple tables using a wildcard
- If the cached results have expired; typical cache lifetime is 24 hours, but the cached results are best-effort and may be invalidated sooner
- If the query runs against an external data source
Quoted from https://cloud.google.com/bigquery/docs/cached-results
