Intel / article

Uber’s AI infra, from the inside

Source: newsletter.pragmaticengineer.com
Author: Gergely Orosz
Date: Published Oct 14, 2026

Context

Uber's engineering team uses a shared model gateway to keep model choice separate from common operational controls. In the Pragmatic Engineer's episode summary with Uber engineer Uday Kiran Medisetty, the platform team decided in 2023 that AI work should not depend on one model or , and built Model Gateway to switch among Anthropic, OpenAI, other vendors and open models Uber runs itself. Interview account.

With that gateway in place, Uber extended it with shared controls: security middleware that checks requests and responses and flags slow models, per-user spend tracking, and audit logs with session traces. Model choices can change without rebuilding those controls. This is Uber's account of its own system, not a measured comparison. Gateway controls.

Key takeaways · AI-distilled

Uber decided in 2023 that its AI work must stay model- and harness-agnostic, so it built a Model Gateway that switches between Anthropic, OpenAI and self-hosted open models, with security checks, per-user spend tracking and audit logs added.
Engineers at Uber get a spend tier, and a manager can approve going past it for the month. A plugin suggests ways to lower an engineer's costs, and the episode says Uber cut token costs by roughly half over the last few months.
Uday Kiran Medisetty calls code review a major bottleneck. Humans still review every PR, but by then AI review has already run, demo screenshots or videos may be attached, and each change carries a risk score to help reviewers decide how much it needs.
CI job volume at Uber more than doubled in a year, prompting an internal code red. The fix moved simple checks such as linting and build rules into the 's local loop, run with an afx validate command before a PR is opened.
Because agents make PR counts easy to inflate, Uber stopped leaning on PR velocity and now labels each PR as feature, bugfix, maintenance, testing or docs. It reports the feature share rising and the maintenance share falling.

Terms in this piece · Glossary

agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

Read the source newsletter.pragmaticengineer.com