Latest from Tencent HY Research blog: Bridging LLM Infra and algorithm development. 🚀
We are open-sourcing GradLoc: A white-box diagnostic tool that traces gradient spikes to the exact culprit token in O(log N) time.
Scaling RLVR no longer has to be a battle against "black-box" gradient spikes. Moving from heuristic trial-and-error to deterministic engineering diagnostics.
Blog: https://t.co/RbxXOmIsyD
GitHub:
🔍 From Global Spikes to Single Tokens
Standard monitoring only tells you when a gradient spike happens. GradLoc tells you where. Using distributed binary search, it isolates the specific token causing the spike with negligible amortized overhead. It’s an "always-on" debugger for LLM training.
💡 New Collapse Mode Discovered
It’s not always "dirty data" or "training-inference mismatch." GradLoc revealed Layerwise Gradient Heterogeneity: Tokens can have "safe" Importance Sampling (IS) ratios but still cause specific layers to explode while others remain stable. Standard global clipping is too blunt a tool for this.
🛡️ The Solution: LayerClip
To address Layerwise Gradient Heterogeneity, we propose Layerwise Gradient Clipping. Instead of a uniform global clamp, LayerClip applies adaptive constraints based on local layer statistics. This fine-grained control stabilizes RLVR training where standard methods fail.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Turns RLVR training instability from guesswork about dirty data into a locatable defect, and shows global gradient clipping can mask per-layer explosions that only local clipping controls.
Read the full thread on X ↗
