Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

A phone feature that performs one narrow task may benefit from a specialized model small enough to run locally. In a talk on tiny models, Google's AI Edge team describes fine-tuning compact models on task-specific examples, including turning text requests into a limited set of device actions.

This approach trades broad, ready-made capability for training focused on the intended job. The speaker reports using it to reach more constrained devices with lower memory needs and faster responses. Choosing a model still requires accounting for the memory used by its runtime and conversation state, as well as its weights. A model file that fits in memory is only part of the device budget.

“we need to really think a lot about kind of quantization and we also really need to think about what is the smallest possible um model we can use for a given task”

Cormac Brick

“we're still at a point where small models are too big because they can't reach like older laptops or kind of more consumer edge devices”

Cormac Brick

“this model knows about 10 different output functions and can call them at over 86% uh reliability from a given arbitrary text input”

Cormac Brick

“in the range of 10,000 to 10 million samples of synthetically generated um data will be sufficient to fine-tune a smaller model to a really really high degree of reliability”

Cormac Brick

“tiny models will will enable reach to a much much larger pool of devices”

Cormac Brick