Vibeleaderboard
← All Intel
Intel / article

Inference engineering for software engineers

Source
newsletter.pragmaticengineer.com
Author
Gergely Orosz
Date

Context

Gergely Orosz's Pragmatic Engineer newsletter lays out what happens when a company moves its AI workload off OpenAI or Anthropic APIs and onto models, models whose trained parameters are published so anyone can host them (Pragmatic Engineer). The motive is usually cost; the article opens with savings it covered earlier, including Uber cutting per-token costs by 50% and Pinterest by 90%. Drawing on interviews with three inference providers, Modal, Fireworks and Baseten, it describes the move as a path in stages rather than a single switch.

Inference providers run open models on their own GPU fleets and sell access through APIs that copy the format of OpenAI's and Anthropic's, so the first stage is often just pointing an application at a new address. That address is shared with other customers, and the article says the per-token price falls while the familiar problems of shared infrastructure remain. One is time to first token, the wait before a reply starts arriving. It has a long tail, with a significant share of requests taking twice the median or longer, because a request can be batched with a more expensive one, can queue while a new copy of the model starts up, or can land on a copy that lacks the cached work from earlier turns of the same conversation. The second stage, renting dedicated GPUs from the provider, gives predictable latency and lets a team keep a chosen model running and it, but the article says GPU scarcity usually makes it an expensive, long-term budget commitment. The providers told Orosz that customers typically reach even the first stage only once their product has traction and token bills have grown.

The same open model is not the same product at every provider, because each one compresses weights, caps or changes defaults. Artificial Analysis's endpoint accuracy index, which compares a provider's results with Artificial Analysis's own hosted copy, ranged from 93% to 102% across providers serving Kimi K3. The article therefore favors a team's own evaluation set over public benchmarks, and compares cost per completed task rather than per token, since models spend different numbers of tokens on the same job. Its example from Artificial Analysis: $5.46 per task for Anthropic's Sonnet 5.5 against $2.00 for Kimi K3. Fireworks' Ember-1, a Kimi K3 derivative trained to reach similar quality with about 40% fewer reasoning tokens, is the article's illustration of providers competing on efficiency.

Moving to a provider does not remove the reliability problem either. The article puts Anthropic's self-declared API uptime at 99.5% and OpenAI's at 99.97%, below the 99.99% expected of cloud providers, but notes that inference providers go down too, so companies commonly route requests through a gateway that falls back to another provider or a frontier API. This explanation covers only the article's free portion, which stops early in the first stage; its sections on configuring dedicated GPUs and on are for paid subscribers.

Terms in this piece · Glossary
  • open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
  • fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
Read the source newsletter.pragmaticengineer.com
More from Gergely Orosz
Recommended reads
Comments

Checking sign-in…

Loading comments…