Firestore 500+ Writes Per Second to One Collection

Question

Firestore has a limit of 500 writes per second to

"a collection in which documents contain sequential values in an indexed field"

https://cloud.google.com/firestore/quotas#writes_and_transactions

What scenarios can I increase the limit to 10,000 writes per second in ONE collection?

Assuming I use Best Practices of keeping Document IDs distributed relatively evenly throughout the key range.

A) What if the documents do not contain any sequential values?

E.g. the only fields I have in each document is below.

• Field A: random string
• Field B: random string

B) What if the document contains a sequential field, but I don't query by that field ever?

• Field A: random string
• Field B (Sequential): date

C) What if the document contains a sequential field, but I turn off indexing to the date in Field B?

• Field A: random string
• Field B (Sequential): date (automatic indexing turned off)

C) Part 2 - If I turn off indexing to that date field (and this removes the 500 per second limitation), what happens if I randomly add a Field C to a SINGLE document in the collection? Does having this one document with a Field C ruin the 10,000 per second write limitation to the collection? Because now Firestore has to worry about automatic indexing of any new Field C?

(and then does this happen every single time you add a random field to any document in this collection, and therefore have to turn off indexing to every possible field in the collection to remove the 500ps limitation?)

Answer 1

500 writes per second is a conservative estimate of what a single server can handle. As Cloud Firestore does horizontal row sharding for both document and index storage; this means documents written with sequential ids, or index entries being added sequentially will be handled by a single server. Putting these together (500 writes per server + sequential updates handled by 1 server) is what gives you that limit.

When you have well distributed data, our horizontal row sharding can split for workload into many servers. For Cloud Firestore in Native mode, this will work until you hit the upper limit of our realtime system, which is around 10K writes/second. For Cloud Firestore in Datastore mode, there is no practical limit as long as you give us a heads up (customers can exceed 1M writes/second)

In short, A) and C) will be limited by the upperbound of the mode (10K or available capacity). B) will be limited by the load a single server can handle.

For C2), there will be no impact if you are just adding it to a single document. The limit is purely based on the incoming load and what servers it gets assigned to.

Answer 2

For the community: Firestore updated their documentation to include more detail into why there is a limit on 500 per second writes, and also a suggested solution on how to overcome it.

https://firebase.google.com/docs/firestore/solutions/shard-timestamp

Solution

1. Add a shard field alongside the timestamp field. Use 1..n distinct values for the shard field. This raises the write limit for the collection to 500*n, but you must aggregate n queries.
2. Update your write logic to randomly assign a shard value to each document.
3. Update your queries to aggregate the sharded result sets.
4. Disable single-field indexes for both the shard field and the timestamp field.
5. Delete existing composite indexes that contain the timestamp field. Create new composite indexes to support your updated queries. The order of the fields in an index matters, and the shard field must come before the timestamp field. Any indexes that include the timestamp field must also include the shard field.

Warning (not highlighted in the docs)

They don't call it out specifically in the docs, but shards may increase the number of reads you have to do. As you can see in Step 1 you must aggregate across shards.

E.g. in the example there are 3 shards, and the limit is 5. The query returns up to 15 documents, but you only want 5. So it slices it back down to 5 at the end.

This would be particularly problematic if you have [10] shards (5000 writes per second) and want the limit to be [50]. You would be aggregating across [500] docs when you only want [50].

Understanding the write for limit sequential indexed fields

The limit on the write rate for sequential indexed fields comes from how Cloud Firestore stores index values and scales index writes. For each index write, Cloud Firestore defines a key-value entry which concatenates the document name and the value of each indexed field. Cloud Firestore organizes these index entries into groups of data called tablets. Each Cloud Firestore server holds one or more tablets. When the write load to a particular tablet becomes too high, Cloud Firestore scales horizontally by splitting the tablet into smaller tabletsand spreading the newtablets` across different Cloud Firestore servers.

Cloud Firestore places lexicographically close index entries on the same tablet. If the index values in a tablet are too close together, such as for timestamp fields, Cloud Firestore cannot efficiently split the tablet into smaller tablets. This creates a hot spot where a single tablet receives too much traffic, and read and write operations to the hot spot become slower.

By sharding a timestamp field, you make it possible for Cloud Firestore to efficiently split workloads across multiple tablets. Although the values of the timestamp field might remain close together, the concatenated shard and index value give Cloud Firestore enough space between index entries to split the entries among multiple tablets.