The Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO
- Source
- youtube.com
- Author
- Latent Space
- Date
Context
Serving an AI model involves different kinds of computing work, which need not all run on the same hardware. Cerebras's technical chief argues for combining specialized systems within a data center, assigning different parts of the workload to hardware suited to them. The design problem becomes choosing the right mixture of computing, memory and communication capacity.
The CTO says assembling that mixture may be unjustified for a small deployment but worthwhile at very large scale. This makes the surrounding system important when judging a fast AI chip: its value depends on how it fits the workload and the rest of the infrastructure. The speaker leads a company that sells its own chips, so the argument is an industry insider's view. See the architecture discussion.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
videoDevDay 2025: Apps SDK, Agent Kit, MCP, Codex and why Prompting is More Important than EverLatent Space
videoThe Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold XinLatent Space
videoThe Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTOLatent Space
Checking sign-in…
Loading comments…



