BigQuery quota limits from query table append apply or not?

Question

In my C# app I use a BigQueryClient.CreateQueryJobAsync to regularly append to a partitioned table table1. Currently this is happening only up to a 50-100 per day.

await BqClient.CreateQueryJobAsync(
    ...,
    new QueryOptions
    {
        DestinationTable = "table1",
        WriteDisposition = WriteDisposition.WriteAppend
    })

I understand that Big Query has many limits and these are usually well documented. But for this particular scenario I am not sure if the limits apply or not. The quotas page says there are "1,000 updates per table per day" but the documentation also explicitly lists which operations are affected by quota. Assuming there is an explicit list there must also be a list of "everything else" where the quota does not apply. For instance, "classic UI" is under the quota which should imply that the "new UI" is not. Similarly, the page states that jobs.query API is affected by quota but since I am using the official C# driver, it leaves me wondering as to whether this applies to my scenario or not.

Apparently, I could write a script to try to do the append operation 1001 times in 24 hours and see whether I hit the quota but I wish I could simply read this from documentation and understand without any ambiguity.

Does anyone know from first-hand experience how this actually works?

Yun Zhang Yun Zhang · Accepted Answer · 2019-10-02T07:57:16

BigQuery C# library internally uses jobs.query API, so the limit/quota applies in your case.

One more thing to be aware of is, since you're writing to a partitioned table, below quota also applies:

Maximum number of partition modifications per day per table — 5,000

You are limited to a total of 5,000 partition modifications per day for a partitioned table. A partition can be modified by using an operation that appends to or overwrites data in the partition. Operations that modify partitions include: a load job, a query that writes results to a partition, or a DML statement (INSERT, DELETE, UPDATE, or MERGE) that modifies data in a partition.

More than one partition may be affected by a single job. For example, a DML statement can update data in multiple partitions (for both ingestion-time and partitioned tables). Query jobs and load jobs can also write to multiple partitions but only for partitioned tables. BigQuery uses the number of partitions affected by a job when determining how much of the quota the job consumes. Streaming inserts do not affect this quota.

BigQuery quota limits from query table append apply or not?

1 Answers