Fly.io's GPU retrospective describes a mismatch between selling computing hardware and what its app-developer customers wanted to buy. The company expected developers to run AI models on rented GPUs; it concluded that many primarily wanted access to language models through APIs. Meanwhile, separating GPU workloads onto dedicated servers reduced utilization and made the infrastructure less economical. In the retrospective, Fly.io says it would scale back ambitions for the product while continuing the existing service. Its experience shows how growing demand for AI can still leave a particular infrastructure offering without enough customers.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
