• The scoreboard in quadratic weighted kappa (agreement with the true label, credited partially for near misses): best LLM 0.578, ConvBERT 0.625. A small encoder trained on the task beats a frontier model prompted for it.
• Both learned models also beat the hand-engineered feature baseline, so the story isn't neural versus classical. It's task-specific training versus general-purpose prompting.
• Why prompting fails here: projecting the item embeddings down to a low-dimensional map, easy and hard items sit on top of each other. Difficulty barely registers in the text's meaning, so a model reading meaning has little signal to read.
• No prompting configuration rescued it. Across several models and strategies the winner was plain zero-shot at temperature 0, meaning the ceiling was the approach, not the prompt.

• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
• calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
• benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
• embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.

If you're using LLMs as judges or graders for difficulty/quality estimation tasks, this shows that scaling to more capable models (GPT-5 vs GPT-4.1) can actually degrade calibration on hard cases, and that a purpose-built encoder model still beats zero-shot LLM prompting on this benchmark.

• articleWhat do LLMs think when you don't tell them what to think about?Together AI
• articleControlling Reasoning Effort in LLMsSebastian Raschka, PhD
• articleUnderstanding the 4 Main Approaches to LLM Evaluation (From Scratch)Sebastian Raschka, PhD

Checking sign-in…

Loading comments…