Vibeleaderboard
← All Intel
Intel / post

Why Reserved Tokens-Per-Hour Inference Pricing Is Broken

Source
x.com
Date
dax@thdxr

most inference companies have settled on "reserved tokens per hour" pricing that means you're agreeing to pay X dollars per hour no matter how much you're actually using if you go over they bill you at a much higher rate. if you go below you're still paying for it. and the provider still sells the capacity to someone else anyway it's so stupid, it's the opposite of why you use the cloud. they're supposed to be spreading supply across many customers and offer flexible pricing this is was the whole breakthrough with AWS - they could charge you per minute for exactly what you used so you could scale up and down inference needs to get there. it's difficult but that is the job. if you can't do it you should die

Context

dax, who builds the OpenCode, argues that the way model-serving companies have started selling capacity puts the risk on the wrong side of the deal. Under what he calls "reserved per hour" pricing, a customer commits to paying a set number of dollars every hour for a reserved amount of model , whether or not it is used. Use more than the reservation and the extra is billed at a much higher rate. Use less and the reservation is still charged in full. He says most companies have settled on this model, though the post names no provider and quotes no prices.

His objection is that this undoes the reason people buy cloud services at all. A cloud provider is supposed to pool demand from many customers, so that one customer's quiet hour is absorbed by another's busy one and each can be billed flexibly. In his account, providers still resell reserved capacity a customer is not using, so they collect from the reservation and from the resale while the customer carries the cost of guessing demand. As a hypothetical illustration of that cost: a team that reserves enough for its busiest hour pays for idle capacity the rest of the day, and a team that reserves for its average hour pays the overage rate whenever usage spikes.

He points to AWS as the precedent, describing its breakthrough as charging by the minute for exactly what was used, so customers could scale up and down. His conclusion is that inference has to reach the same kind of usage-based billing. He concedes it is difficult and treats that difficulty as the provider's job to solve, in blunt terms: a provider that cannot do it, he writes, "should die". The post is an opinion from a buyer's side of the market and offers no figures, so it describes a pricing structure and a complaint about it, not measured costs.

Terms in this piece · Glossary
  • Agent mode — A mode in which one agent can inspect, edit, run commands, and iterate toward an outcome.
  • Token — A piece of text, such as a word, part of a word or a punctuation mark, that a language model represents as numbers and handles as a unit.
  • throughput — How much model output a provider can generate over time, commonly measured in output tokens per second.
  • Inference — Using a trained model to produce an output from a supplied input.
More from dax
Recommended reads
Comments

Checking sign-in…

Loading comments…