When Chain-of-Thought Helps and When It Hurts: An Empirical Investigation of the Serial-Depth Bottleneck in LLM Reasoning
Source
arxiv.org
Author
Tughanbulut Kurtulush
Date
Context
Asking a model to write intermediate reasoning helped some tasks much more than others in a study of three models and five benchmarks. The author reports gains of 54–68 percentage points on the two mathematics benchmarks, but a 28.7-point decline for the seven-billion-parameter model on HumanEval coding problems. Research abstract.
The proposed explanation is that intermediate text helps when a solution requires many dependent steps. The measured effects nevertheless vary with both task and model. The paper's architectural interpretation goes beyond the direct observation that the same prompting strategy can produce large gains or losses.
Terms in this piece · Glossary
chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
articleWhy We ThinkLilian Weng
articleThe First Few Tokens Matter: Understanding Chain-of-Thought ControllabilityHugh Van Deventer, Dmitrii Troitskii, Laura Ruis
articleCoT-Core: Accelerating LLM Evaluation via CoT-Aware Coreset SelectionQihua Pan, Zhenheng Tang, Peijie Dong, Xiang Liu, Huacan Wang, Bo Li, Xiaowen Chu