How We Made Our Text-to-Speech API Free: The Inference Engineering Behind S2.1 Pro
- Source
- fish.audio
- Author
- Fish Audio editorial sitemap
- Date

Context
Fish Audio's free S2.1 Pro text-to-speech API, whose terms were covered here earlier, invites an obvious question: how does a company give away a speech model it also sells? In this engineering post, founder Shijia Liao answers with cost rather than subsidy. Fish Audio says the same volume of requests that used to need four NVIDIA H200 GPUs now runs on one, and that the free tier uses the same model weights as the paid one.
The problem starts with how speech models produce audio. They generate it one small token at a time, and for a single request each step is a little arithmetic followed by a wait while the GPU's memory delivers data, so an expensive chip sits mostly idle; the post says a naively served endpoint handling one request at a time can run below 10% utilization. Fish Audio's own GPU code, released as the open-source fish-scales-ops library, stores weights in FP8, an 8-bit number format that halves the data each step must read compared with 16-bit, and merges operations that standard libraries run separately with a trip to memory in between. At these small per-step sizes the post reports 2.1 to 4.3 times the speed of PyTorch's compiled cuBLAS . Continuous batching, adapted from the SGLang serving engine, then lets new requests join the running batch as others finish instead of waiting for a whole batch to end.
The batching figures show where the capacity comes from. On one H200, combined output rose from 154 audio with one request to 8,006 with 64 at once, about 52 times, while the median wait for the first audio rose from 73 to 110 milliseconds. Each request's own rate falls only from 154 to roughly 125 tokens per second, so the chip now serves many listeners in time it used to spend waiting on one. To guard quality, Fish Audio says a serving change ships only if the 8-bit model's first five audio frames match the 16-bit model's token for token on a fixed input, a check that matters because one memory-layout bug during development produced badly wrong outputs while the speed benchmarks looked normal.
Every figure is Fish Audio's internal measurement of its own system, and the post shows no before-and-after serving test for the fourfold claim; it presents that number as the combined effect of kernels, batching, scheduling and owning its fleet of about 500 GPUs along with its network and storage. The library's public README reports kernel and layer timings rather than whole-service throughput. One component also works differently from how the summary counts it: utilization rose from about 50% to more than 90% by filling idle GPU time with preemptible data-cleaning jobs, which puts spare hardware to work on Fish Audio's own data preparation rather than serving more speech, yet the post credits it with roughly halving the cost per request. On terms, the post says free access has no hard usage cap, lasts for an initial period, and will change only with advance notice.
- Fish Audio says TTS decode runs small matrix multiplies (M up to 128) that cuBLAS and torch.compile are not tuned for; its open-source fish-scales-ops FP8 kernels report 2.1-4.3x speedups over torch.compile-fused cuBLAS at those shapes.
- A weight-scale memory layout bug in the MXFP8 path caused about 53% relative logit error at normal kernel speed and collapsed output to one repeating token; it showed up only in output token analysis, not in performance metrics.
- Every -stack change must pass two gates: greedy decode must match bf16 token-exactly on the first five audio frames, and per-request decode throughput must stay within 5% of a frozen baseline at every concurrency level.
- On one H200 with continuous batching, Fish Audio measured aggregate throughput rising about 52x, from 154 tok/s at concurrency 1 to 8,006 tok/s at 64, while median time to first byte rose from 73ms to 110ms.
- Running preemptible data-cleaning jobs at lowest priority on idle GPUs lifted cluster utilization from about 50% to over 90%, which Fish Audio says roughly halves the effective cost per inference request.
- Path — A description of a file or folder location, written using directory names and separators.
- throughput — How much model output a provider can generate over time, commonly measured in output tokens per second.
- quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
- Inference — Using a trained model to produce an output from a supplied input.
Checking sign-in…
Loading comments…






