From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI
Source
youtube.com
Author
AI Engineer
Date
Context
Perceptron describes a way for models processing continuous camera input to choose which visual information deserves computation. In its embodied-AI talk, a routing component selects which portions of the input each model layer reads and which it skips.
That addresses a specific problem for robots: cameras keep producing images, while the details relevant to the current task may occupy only part of the scene. The approach aims to make processing selective according to the task, rather than compressing every region equally. The useful idea is to learn where to spend computation as part of perception itself.
Key takeaways · AI-distilled
Armen Aghajanyan notes that an hour of video yields about a million visual tokens, yet supervision lands on roughly 2% of them; Perceptron's response is a perceptive objective that learns which percepts matter instead of predicting every pixel equally.
To handle context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → bloat from always-on cameras, Perceptron uses a data-sparse mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → whose router decides per layer which tokens to read and which to skip, beyond the ~10x compression patch averaging offers.
Perceptron treats detection as an agentic task: the model tiles the image, raises contrast, and proposes boxes iteratively until it finds the target.
The headline claim is a scaling law: training jointly on perception, reasoning, and control lets 10x more video pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → substitute for 10x less teleoperation data, which he puts at about $100 per hour.
Terms in this piece · Glossary
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
reasoning model — A model trained to think — generating extended internal reasoning before answering — trading time and tokens for accuracy on hard problems.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
videoYour Financial Agent Needs to Show Its Work - Anirudh Veeraragavan, Fieldguide
videoBuild Your Own Finance Agent Inside ChatGPT - Stefano Fabbri & Oliver Steffmann, OpenAI
videoDon't Vibe Code Your Agent Infrastructure - Akshat Bubna, Modal
videoHow to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere
