If you're serving LLMs and using speculative decoding, Aurora replaces one-time offline draft-model training with an RL loop that adapts the speculator to your live request distribution, reportedly yielding 1.25x throughput over a well-tuned static speculator.
Aurora is an open-source RL framework that turns speculative decoding from a one-time offline setup into a self-improving system that learns from every request it serves.
Transcript
1.25x over a well-trained static speculator. Aurora is an open-source RL framework that turns speculative decoding from a one-time offline setup into a self-improving system that learns from every request it serves.
