Where Does a Robot Think – On-Device vs Datacenter Inference
Source
Ivan Chiam
Author
Ivan Chiam
Date
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Anyone building robotics or edge-AI products needs to design the model around serving constraints (latency, cost) first, the reverse of how cloud LLMs are typically built.
Key quotes
“The model comes first; the hardware serves it.”
“Jitter is the biggest enemy.”
“So the real question is not whether robots can get wafers. It is silicon efficiency: for a given fleet of robots, which approach consumes less leading-edge silicon?”
“Serve a VLA fleet with continuous batching; serve a video WAM one robot at a time.”
