Vibeleaderboard
← All Intel
Intel / article

AI Gateway now supports Private Inference

Source
vercel.com
Author
Sam Chitgopekar
Date

Context

Vercel's AI Gateway, a service that puts many hosted AI models behind one API key and one set of logs, can now route requests to models a company runs itself (Vercel changelog). A team registers its own endpoint, such as a fine-tuned model served with vLLM, as a private provider, adds the models it serves, and calls them with an ID like private/support-. The gateway translates each request into the endpoint's format, so code using the Chat Completions, Responses or Messages APIs can switch between a private model and a catalog model by changing only the model ID. Coding agents already connected to the gateway, including Claude Code, Codex and Cursor, can use the private models too.

"Private" here means visible only to the owning Vercel team, not reachable only inside a company network. The documentation requires a publicly reachable HTTPS endpoint and rejects addresses that resolve to private, loopback or link-local networks, so a model running only on an internal network needs a public endpoint before the gateway can reach it. Private models are left out of the public catalog and model listings, credentials are stored encrypted, and Zero Data Retention can be switched on for each provider. The documentation tells teams to enable that setting only if their endpoint keeps no request data, and any member allowed to create a provider can turn it on as an attestation, so it records the team's own claim about its server; the documentation describes no check by Vercel.

Billing works differently from catalog models. Vercel charges no fees for private models, since the team pays for its own servers, but it adds a fee of $0.20 per 1,000 requests unless the request also names a catalog model as a fallback while the default setting that lets Vercel's own provider keys serve fallbacks is on. With that fallback, there is no fee, and a failed private call is served by the paid catalog model instead. Private requests appear in logs, trace drains and observability charts alongside catalog traffic, broken down by requests, tokens and time to first token. OpenRouter launched a comparable Private Endpoints feature in October, which routes a team's fine-tuned models, dedicated deployments and reserved capacity through OpenRouter alongside its public catalog.

Key takeaways · AI-distilled
  • Private registers a team's own model endpoint in AI Gateway as a private provider, using its base URL, API format and credentials, and exposes its models under private/<slug> IDs. The Vercel CLI can do this from version 65.16.0.
  • AI Gateway translates Chat Completions, Responses or Messages requests into the private endpoint's own format, so switching between a private model and a catalog model changes only the model ID.
  • Private models are visible and callable only by the owning Vercel team and are not listed in the public catalog. Credentials are stored encrypted, and Zero Data Retention can be enabled per provider.
  • A request can list catalog models as fallbacks if the private endpoint fails, and private traffic shows up in the same logs, trace drains and charts, broken down by requests, tokens and time to first token per model.
Terms in this piece · Glossary
  • fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
More from Sam Chitgopekar
Recommended reads
Comments

Checking sign-in…

Loading comments…