Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL

When training and response generation run on separate machines, the response server needs updated model weights, the numerical values learned during training. This TRL implementation report describes transferring only values whose stored representation changed, along with their positions. The authors report that in bf16, the compact number format used, roughly 99% of weights are identical between consecutive training steps, so the changes are small. Periodic full snapshots provide a starting point from which a server can apply subsequent updates.

The trainer uploads these changes to shared online storage while the response server continues working, then pauses it to download and apply the update. The authors report a working Wordle training demonstration. Their discussion of much larger models is extrapolation, and they explicitly leave multi-node training measurements for future work.