Error form google big query The query specified one or more federated data sources but not all of them were scanned

0
votes

I recently encountered the following error while working with external tables The query specified one or more federated data sources but not all of them were scanned. The error affects a few models, where as other models with similar syntax run without an issue. The set up is as follows,

  1. I am using hive partitioning specifying that there is a path component arrivaldat:e gs://bucket/mydata/{arrivaldate:DATE}
  2. I am specifying partition pruning arrivaldate between DATE_SUB(CURRENT_DATE(), INTERVAL X DAY) AND CURRENT_DATE()
1
google-bigquery

1 Answers

0
votes

It looks like in my particular case, the issue stems from the fact that, there are no partitions satisfying the range specified by the WHERE clause. Widening the window X to include at least one partition seems to have fixed this issue. As a more generic case as also seen in similar error #57492661, it would appear that the error stems from GBQ not finding any files