Vibeleaderboard
← All Intel
Intel / video

The Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO

Source
youtube.com
Author
Latent Space
Date

Context

Serving an AI model involves different kinds of computing work, which need not all run on the same hardware. Cerebras's technical chief argues for combining specialized systems within a data center, assigning different parts of the workload to hardware suited to them. The design problem becomes choosing the right mixture of computing, memory and communication capacity.

The CTO says assembling that mixture may be unjustified for a small deployment but worthwhile at very large scale. This makes the surrounding system important when judging a fast AI chip: its value depends on how it fits the workload and the rest of the infrastructure. The speaker leads a company that sells its own chips, so the argument is an industry insider's view. See the architecture discussion.

Terms in this piece · Glossary
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from Latent Space
Recommended reads
Comments

Checking sign-in…

Loading comments…