Ai2 allocates GPU time through team budgets, giving jobs from teams below their share priority over those above it. Usage is tracked over a rolling seven-day window, so a team can catch up after a quiet period. Spare capacity remains available for interruptible work without spending budget. Its engineering account reports occupancy holding at 98% before and after the change over a 30-day test, with 18% of delivered GPU time unallocated: sharing was organized around funded priorities while the machines stayed busy.
Key takeaways · AI-distilled
Ai2 replaced its priority scheduler after seeing GPU squatting with no-op jobs, priority inflation until 100% of jobs ran HIGH, and on-call time spent negotiating shutdowns of non-preemptible jobs on faulty hosts.
Manager-set GPU-time budgets feed a hierarchical fair-share scheduler with a 7-day lookback that sorts under-served allocations first. Unbudgeted time keeps GPUs busy but can be preempted by any funded request; it was 18% of delivered GPU time.
Each job declares a minimum runtime, capped at 8 hours, during which it cannot be preempted; afterward resumable work can be requeued. Letting unhealthy hosts drain this way cut repairs needing a human by 74%.
Over a 30-day test, teams received 98% of the GPU hours they were owed while occupancy held at 98%. Debug-job p90 queue wait fell from 2 hours to 30 seconds, though Ai2 notes the small baseline sample adds variance.
Costs remain: interactive dev sessions lost week-long holds, prompting plans for a CPU-only cluster and restorable sessions, and Ai2 is investigating whether minimum-runtime protection fragments capacity for the largest jobs.
