Intel / article

CoT-Core: Accelerating LLM Evaluation via CoT-Aware Coreset Selection

Source: arxiv.org
Author: Qihua Pan, Zhenheng Tang, Peijie Dong, Xiang Liu, Huacan Wang, Bo Li, Xiaowen Chu
Date: Published Aug 4, 2026

Context

-Core selects a smaller set of test questions by comparing generated solution paths. The idea is that questions with different wording may require similar reasoning, so a useful subset should represent those reasoning patterns. The method asks a to produce reasoning traces, converts them into numerical representations and groups similar traces. Method abstract.

The authors report lower evaluation costs while preserving score estimates across four benchmarks. They also find that effectiveness depends on task complexity. The contribution is a way to choose which questions to run during repeated model evaluation, with reported benefits that vary by task. Reported findings.

Key takeaways · AI-distilled

CoT-Core picks evaluation coresets by clustering questions on their zero-shot chain-of-thought trajectories projected into latent space - grouping by underlying logic rather than surface text similarity.
It is training-free and needs no historical response logs, sidestepping the cold-start bottleneck that blocks Item Response Theory-style coreset methods.
On GSM8K, MMLU, MMLU-Pro and GPQA it sharply cuts evaluation cost while keeping score estimates high-fidelity.
Boundary condition worth noting: the efficacy of reasoning-aware pruning is gated by task complexity - it is not a uniform win across benchmarks.

Terms in this piece · Glossary

chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.

Key quotes

“We propose CoT-Core, a novel training-free core question selection framework.”

“Recognizing that lexically disparate questions can share equivalent underlying logic, CoT-Core prompts LLMs to unroll zero-shot Chain-of-Thought (CoT) reasoning trajectories.”

Read the source arxiv.org