Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google
Source
AI Engineer
Author
AI Engineer
Date
Terms in this piece · Glossary
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
Names the actual bottleneck for on-device models — DRAM cost, which is getting worse, not better — and shows the quantizationShrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.Full definition → and fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → numbers that decide what fits.
Key quotes
“we need to really think a lot about kind of quantization and we also really need to think about what is the smallest possible um model we can use for a given task”
Cormac Brick
“we're still at a point where small models are too big because they can't reach like older laptops or kind of more consumer edge devices”
Cormac Brick
“this model knows about 10 different output functions and can call them at over 86% uh reliability from a given arbitrary text input”
Cormac Brick
“in the range of 10,000 to 10 million samples of synthetically generated um data will be sufficient to fine-tune a smaller model to a really really high degree of reliability”
Cormac Brick
“tiny models will will enable reach to a much much larger pool of devices”
