Regarding the execution time inconsistency, this does seem to be a higher variance than I would expect. Can you provide a job id of a fast query and a slow query so I can look up where the time was being spent in the internal query statistics?
That said, some fairly significant variation in query times, while not quite on the range of what you're seeing, is unsurprising. Here are some of the factors:
Tail latency. The query is broken up into pieces and farmed out to several different workers (potentially thousands, depending on the size of your data). The data is being read from a distributed filesystem cluster that will likely have your data striped across hundreds of disks or more (depending, again, on the size of your table).
The slowest one of these components to respond will determine your total query time. This is called tail latency, meaning that you have to wait until the long tail of stragglers is finished. We do a lot of work to try to minimize the effect, replicating data and re-dispatching work, but it can still have a big effect.
Load. Currently when our clusters are heavily loaded, it can slow down response times for other users. We're working on much better isolation mechanisms, but they're still a little ways out. This wouldn't account for time discrepancies of the magnitude that you're seeing, but it can be a factor.
Throttling. When a single customer sends multiple parallel queries at once, those queries may be slowed down in order to prevent that customer from taking up too much capacity. How much and whether this happens depends on a number of factors, including query size and the other load on a cluster.
Writing results. If your results are larger than 100k or so, writing the results out may be very slow, and may have absurd variation. This is a bug that we're currently investigating.
There are significant efforts under way to reduce the impact of all of these factors. Right now, however, we don't have a magic wand that we can wave and say "query performance is going to be consistent to within 20%", other than saying "we recognize the issue and are working on improving it".
If you provide job ids, we can look at the specific cases of your queries to figure out where the time is being spent and if there is something we can do to address the issue.
