Large clusters for small models — Daniel Svonava, Superlinked
Source
youtube.com
Author
AI Engineer
Date
Context
Running many specialized models creates a scheduling problem: the system must keep assigning short requests as computing capacity becomes available. In its presentation, model-serving company Superlinked describes replacing a central router's assignments with a shared queue from which workers take their own work.
The published account says workers also assemble their own batches, grouping requests for processing together. The motivation is that a central router's picture of worker availability can become stale. Letting workers pull requests moves the scheduling decision closer to the machines doing the work, addressing how a fleet of small models stays busy.
Key takeaways · AI-distilled
A single two-to-three-generation-old GPU turns roughly 500,000 tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →/second into embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → in the low tens of milliseconds, versus a managed endpoint that costs orders of magnitude more and takes hundreds of milliseconds.
Svonava's example: a contract-review AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → ends up running nine different small models, each chosen for a narrow task -- e.g. the model that has seen the most Vietnamese receipts wins Vietnamese receipt OCR.
Two specific failure modes pushed Superlinked to build its own stack: open-source serving tools ship untuned (adopting one becomes a research project), and ad hoc LoRAs and overnight fine-tunes turn every deployment into a running negotiation between AI and infra engineers.
The open-source (Apache 2.0) stack uses a Rust sidecar that abstracts fifty adapters across three runtimes, plus an autoresearch loop that ships every model already tuned.
One concrete output: an 80-cent LoRAA cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.Full definition → that lifted retrieval on German legal text by 18%.
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.
videoWhy AI Makes You 100x Faster but Not Your Company - Mitchell Troyanovsky, Basis
videoBuilding the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest
videoDon't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse
videoWithout Caching, Your AI Agent Costs 10x More - George Cameron, Artificial Analysis
