What Runs Where? Local AI in Production — Ahmad Osman, Osmantic
Source
youtube.com
Author
AI Engineer
Date
Why it matters
Explains how to match hardware, inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → engine and quantizationShrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.Full definition → to AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → workloads, and how GPU ownership costs compare with renting, so you can judge when local models are viable.
Terms in this piece · Glossary
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.