Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
Source
Ronglong Bao
Author
Ronglong Bao
Date
Key takeaways · AI-distilled
Converting 21 of 28 attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → layers of Qwen3-0.6B to KDA linear attention on a consumer GPU gets perplexity close to the teacher - yet multiple-choice accuracy stays near random (25-29% vs 50.6%), a failure distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition → metrics cannot see.
A four-permutation diagnostic that rotates answer options exposes the cause as interface injury: the model predicts label A 81% of the time regardless of content.
A 1,000-step format-targeted completion-only KL stage repairs the interface (+12.48 points on C-evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →, label-stickiness roughly halved); persona SFT and one DPO round then preserve scores.
Engineering gotcha worth stealing: an FP32-master failure mode where bf16 optimizer updates are silently swallowed - the kind of silent no-op that stalls convergence at small budgets.
Terms in this piece · Glossary
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Converting attention layers to linear KDA attention can silently break a model's ability to answer content-dependent questions even while perplexity looks fine; a permutation-based diagnostic exposes this and a targeted completion-only KL fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → stage repairs most of the damage, offering a checklist for anyone doing similar attention linearization work.
Key quotes
“an interface injury that standard distillation metrics cannot see”
“distill the engineering lessons -- including an FP32-master failure mode in which bf16 optimizer updates are silently swallowed -- that made convergence possible at this budget”
