Large clusters for small models — Daniel Svonava…

Key takeaways · AI-distilled

A single two-to-three-generation-old GPU turns roughly 500,000 /second into in the low tens of milliseconds, versus a managed endpoint that costs orders of magnitude more and takes hundreds of milliseconds.
Svonava's example: a contract-review ends up running nine different small models, each chosen for a narrow task -- e.g. the model that has seen the most Vietnamese receipts wins Vietnamese receipt OCR.
Two specific failure modes pushed Superlinked to build its own stack: open-source serving tools ship untuned (adopting one becomes a research project), and ad hoc LoRAs and overnight fine-tunes turn every deployment into a running negotiation between AI and infra engineers.
The open-source (Apache 2.0) stack uses a Rust sidecar that abstracts fifty adapters across three runtimes, plus an autoresearch loop that ships every model already tuned.
One concrete output: an 80-cent that lifted retrieval on German legal text by 18%.

Terms in this piece · Glossary

token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.

Why it matters

Superlinked's Daniel Svonava describes routing requests to many small, task-specialized models via a shared queue where workers self-batch, doubling cluster throughput compared to top-down routers that stall near 30% utilization under fragmented small-model traffic.

Intel

Large clusters for small models — Daniel Svonava, Superlinked