A local stack splits in two: an inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → server (Ollama, vLLM, LM Studio, MLX) that serves the weights, and a agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → that gives the model file reads, edits, shell commands, and verification. Either side swaps independently.
Models get tuned against their maker's own harness. Nvidia's Polar paper benchmarked Qwen3.5-4B across coding harnesses and it scored best inside Qwen-Code, both before and after RL training, so pairing model to native harness is free accuracy.
Qwen3.6 35B-A3B is a roughly 22 GB download that wants 30-40 GB of RAM, and it runs at usable speed on a Mac Mini M4 or a DGX Spark. The floor for a local coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → is a well-specced desktop, not a GPU cluster.
On Apple Silicon, pull the *-mlx build of a model: it is compiled against Apple's Metal shaders. Linux boxes take the plain tag. Same weights, different runtime, meaningfully different speed.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Most agent tutorials assume you're piping code to a frontier API; Raschka instead wires up the whole loop locally — open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → models served through an inference runtime, driving a real coding agent harness. If you care about keeping code off third-party servers, cutting API bills, or just understanding how the harness-to-model plumbing actually fits together, this walks you through a working stack you can stand up yourself.
Key quotes
“As we can see, Claude Code uses by far the most tokens on average, Codex the least.”
“For example, one Claude run used about 578k input tokens but only about 4.5k output tokens across 25 turns.”
