← All IntelClip / Developer ToolsSharding as the next step, and how it differs from a replica set
From Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face · ≈16:39
Names the exact scaling ceiling a replica set hits and what sharding buys — CPU, memory, storage, reads and writes all scale horizontally, at the cost of a non-trivial shard-key decision.
What’s in it
- Names the exact scaling ceiling a replica set hits and what sharding buys — CPU, memory, storage, reads and writes all scale horizontally, at the cost of a non-trivial shard-key decision.
Clip transcript
However, with 14 million users, 3 million models and our grow soon single MongoDB replica set will not be enough. The next step is sharding. Sharding means scaling your database horizontally. Instead of putting full data set on one replic on one replica set cluster, we are going to cut data into pieces and put each piece on separate chart. Each shard will have his own replication primary and secondary. So we will keep replication just multiplied. The key difference between replica set cluster and sharded cluster is replica set keep full data set on each node. Sharded cluster keep only part of the data on each shard. And then if you want to scale horizontally more you are just adding more shards and then MongoDB balancer will balance data across all those shards. There is also shard key which must be selected. This is not trivial operation but this talk is not about choosing short key. This way we are going to scale everything CPU memory storage reads and writes.
Comments
Checking sign-in…
Loading comments…