Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
Source
Ronglong Bao
Author
Ronglong Bao
Published
Why it matters
Converting attention layers to linear KDA attention can silently break a model's ability to answer content-dependent questions even while perplexity looks fine; a permutation-based diagnostic exposes this and a targeted completion-only KL fine-tuning stage repairs most of the damage, offering a checklist for anyone doing similar attention linearization work.
This paper converts most attention layers of Qwen3-0.6B-Base into linear KDA attention and finds that standard distillation hides a hidden failure: the student model collapses to guessing option 'A' regardless of content, despite near-teacher perplexity.
The authors introduce a permutation-based diagnostic to expose this 'interface injury' and show a targeted completion-only KL fine-tuning stage largely repairs it, recovering +12.48 points on C-Eval.
Transcript
arXiv:2608.02689v1 Announce Type: new Abstract: We convert 21 of 28 full-attention layers of Qwen3-0.6B-Base into KDA (Kimi Delta Attention) linear-attention layers on a single consumer-grade GPU budget, and ask a simple question: what exactly does the conversion break? After surgery, hidden-state alignment and end-to-end KL distillation drive the student close to its teacher in perplexity, yet multiple-choice accuracy stays near random chance (25-29% vs. the teacher's 50.6% on C-Eval). Using a four-permutation diagnostic that rotates answer options while holding content fixed, we show the model sticks to option labels (predicting "A" 81% of the time; 106/161 questions keep the same label under all four rotations) rather than following answer content -- an interface injury that standard distillation metrics cannot see. A 1,000-step format-targeted completion-only KL stage repairs the interface (+12.48 points on C-Eval, label-stickiness roughly halved), after which persona SFT and one round of on-policy DPO preserve benchmark scores within noise. We release code, weights, recipes, and the full audit trail, and distill the engineering lessons -- including an FP32-master failure mode in which bf16 optimizer updates are silently swallowed -- that made convergence possible at this budget.
