Intel / article

Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model

Source: arxiv.org
Author: Ronglong Bao
Date: Published Aug 5, 2026

Context

A modified can resemble its original on text prediction while failing to select answers correctly. After replacing most layers in Qwen's Qwen3-0.6B-Base, a researcher found near-chance multiple-choice accuracy despite a text-prediction measure approaching the original model's. Rotating the answer options exposed the problem: the modified model often kept choosing the same letter even when the answer content moved. Experiment.

A further training stage targeted the answer format and improved C-Eval accuracy by 12.48 percentage points, while roughly halving this attachment to option labels. The experiment shows why checking the model's actual response format can reveal damage that a broader text-prediction score misses. Repair result.

Key takeaways · AI-distilled

Converting 21 of 28 attention layers of Qwen3-0.6B to KDA linear attention on a consumer GPU gets perplexity close to the teacher - yet multiple-choice accuracy stays near random (25-29% vs 50.6%), a failure metrics cannot see.
A four-permutation diagnostic that rotates answer options exposes the cause as interface injury: the model predicts label A 81% of the time regardless of content.
A 1,000-step format-targeted completion-only KL stage repairs the interface (+12.48 points on C-Eval, label-stickiness roughly halved); persona SFT and one DPO round then preserve scores.
Engineering gotcha worth stealing: an FP32-master failure mode where bf16 optimizer updates are silently swallowed - the kind of silent no-op that stalls convergence at small budgets.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.

Key quotes

“an interface injury that standard distillation metrics cannot see”

“distill the engineering lessons -- including an FP32-master failure mode in which bf16 optimizer updates are silently swallowed -- that made convergence possible at this budget”

Read the source arxiv.org