← All IntelClip / Developer ToolsPrecompute the tokens at write time, not at query time
From Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face · ≈7:56
A directly reusable search technique: splitting model names into a stored token array on insert moves the cost off the hot read path and makes autocomplete instant.
What’s in it
- A directly reusable search technique: splitting model names into a stored token array on insert moves the cost off the hot read path and makes autocomplete instant.
Clip transcript
Now let's check how search works in details. For example, someone wants to search the model on the hub and let's say that's llama. So someone type llama into hugging face search bar. His request flows through the hub to an optimized read collection on MongoDB. And this is not our main repo collection when we keep all the data. It's a separate denormalized copy only for reads and listings. The key information is on the left. We tokenize model names on insert time, not at query time. For example, someone wants to publish model meta minus llama/ llama 3.18b. We split the long model name into small tokens like meta lama 3.18b and we store them in an array in MongoDB document. Next, Atlas search which is using Apache Lucin under the hood use autocomplete to find matching models instantly.
Comments
Sign in to comment.
Loading comments…