Few-Shot Degradation Is Not What It Seems: Behavioral Evidence, Representation Analysis, and a Random-Text Control Across 12 Models, 2 Tasks, and 2 Architectures
Source
arxiv.org
Author
Volodymyr Ovcharov
Date
Context
Adding examples to a prompt changes both its content and its length, which complicates attempts to explain why examples sometimes hurt performance. This study used equally long random text as a control, then subtracted the associated change in models' internal representations. Across its Ukrainian news-classification and legal-outcome tasks, the remaining content-related change was more strongly associated with improvement than the raw change. This challenges the interpretation that larger internal changes necessarily mean examples are disrupting the model: some changes may reflect useful learning from the examples.
Key takeaways · AI-distilled
Volodymyr Ovcharov tested 12 open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → models on Ukrainian news classification and legal case prediction. Giving examples in the prompt, called few-shot prompting, helped much more on news: +24 percentage points versus +3.4 on legal text, with two models worsening.
Prompts with examples are longer than prompts without them. The study compared examples against random text of the same length, separating changes caused by extra text from changes caused by useful example content.
The resulting content-delta metric was more closely associated with whether examples helped or hurt performance (correlation 0.65) than the raw change in the model's internal representation was (0.20).
Masking demonstrations in Llama 3.3 70B causally confirmed that models which restructure their representations more based on demonstration content benefit more, the opposite of the intuitive idea that more distortion means more disruption.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
articleWhen Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language ModelsYinfeng Wang, Zhiyuan Yao, Zheren Fu, Lei Zhang, Zhendong Mao
articleLatent Undertow: How Ordinary Typos Break ProbesElad David, Max Fomin, Amit LeVi
articleWhere Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 LossZhiyuan Wang