Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI
Source
youtube.com
Author
AI Engineer
Date
Why it matters
Explains why the same open-weight model performs differently across inference providers on long, repetitive agent inputs, and why reliability is part of effective speed.
Key takeaways · AI-distilled
FriendliAI's Yunmo Koo argues open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → models now compete with top proprietary ones, showing GLM 5.2 and Claude Opus 4.8 build similar tower defense games from one prompt, with GLM 5.2 more than 5.6x cheaper, per the vendor's demo.
Running the same model and coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → on two providers, FriendliAI built a game in about two minutes versus more than three for the other provider. Koo attributes the gap to serving choices, not the model.
Koo cites prefix caching with cache-aware routing, a GPU, CPU and NVMe KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → hierarchy for long repetitive agent inputs, and hybrid speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → that adapts to live traffic. He treats reliability as part of speed, since one failed call can sink a run.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
videoHow Secrets Leak Through AI Agents (and How to Stop It) — Venice AI
videoFrom Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
videoBuild a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
videoContinuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
