Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
- Source
- huggingface.co
- Date
Context
Choosing which processing blocks to remove from a is a problem of combinations: a block's importance can change when another block is removed. The authors of this compression study estimate both individual contributions and pairwise interactions from a small dataset, then search for a set of deletions that preserves model quality.
The resulting calculation resembles an Ising model from physics, with each binary variable representing whether a block stays or goes. Once the interaction estimates are computed, candidate deletion patterns can be scored cheaply without running each through a full model . The score produces promising candidates for testing; the authors report that its mathematically best configuration is not always the best-performing compressed model.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleWhen Planning Problems Become Hard for LLMs: Separating Structural Complexity from Semantic InterpretabilityMayukh Debnath, Adeyinka Adedigba, Rammohan Mallipeddi
articleBBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word ProblemsYutaro Yamada, Kei Hiroshima, Nozomu Yoshinari, Kento Uchida, Shinichi Shirakawa
articleA Removal Based Approach to Improve LLM Faithfulness at Test-TimeQinglan Luo, S M A Nahian, John Guttag, S. Mazdak Abulnaga, Katie Matton
Checking sign-in…
Loading comments…