The MongoDB documentation says that sharding should be employed when one server/replica is insufficient to store all the data.
Given that a dataset that can be scaled to both 100GB & 1GB and executing the same queries on both the datasets, can we say that -
Sharding 100GB across 5 shards of 20GB each be equivalent to sharding 1GB across 5 shards of 200MB each. Would the scale factor effect the way sharding is carried out by Mongo? If yes, where will the changes be observed?
Given that a dataset that can be scaled to both 100GB & 1GB and executing the same queries on both the datasets, can we say that -
Sharding 100GB across 5 shards of 20GB each be equivalent to sharding 1GB across 5 shards of 200MB each
From a high-level view, the sharded cluster architecture could be similar in both of your examples: 5 shards, 3 config servers, and some number of mongos processes. I would hesitate to call that "equivalent" in the same way that a moped is not equivalent to a motorcycle, although both are two-wheeled vehicles in this analogy so interpretation depends on your viewpoint.
However, it's certainly possible to start with a 5 shard cluster provisioned with resources (RAM/CPU/Storage) to meet a certain expected workload and later upgrade (or downgrade) the same cluster with resources to match changing requirements for your use case.
Would the scale factor effect the way sharding is carried out by Mongo? If yes, where will the changes be observed?
The main behaviour difference based on the volume of sharded data will be the sharded cluster balancing activity. Balancing is based on chunks, which are logical contiguous ranges of shard key values that represent roughly 64MB of data by default.
Balancing of chunks between shards is triggered using a migration threshold based on the difference between the shards with the least and most chunks as well as the total number of chunks in a sharded collection:
| Number of Chunks | Migration Threshold |
|====================|=====================|
| Fewer than 20 | 2 |
| 20-79 | 4 |
| 80 and greater | 8 |
With only 100MB of data per shard, that would be roughly 2 chunks (or ~10 overall).
With 20GB of data per shard, there would be at least 312 chunks per shard (likely more, as the chunks are preemptively split rather than always being full).
If you choose a good shard key which effectively distributes data across shards, re-balancing should not be required frequently. A poor shard key, on the other hand, will require more frequent balancing and problems will be much more evident at scale because of the extra I/O overhead.
