Intel / article

Can LLMs Really Understand Item Difficulty Levels? Implications for Automated Item Generation Using LLMs

Source: arxiv.org
Author: Xinyi Wang, Hong Jiao, Ming Li, Sydney Peters, Hanna Choi, Tianyi Zhou, Qingshu Xu
Date: Published Aug 3, 2026

Context

A model's ability to answer a test question does not establish that it can judge how difficult the question will be for test takers. A study of Reading and Writing assessment items found that the tested language models struggled to label hard questions. The best language-model configuration, OpenAI's GPT-4.1, achieved lower agreement with difficulty labels than ConvBERT, another used for comparison. GPT-5.4 tended to underestimate difficulty. For systems generating questions at a requested difficulty level, the finding identifies an gap: the generated question's difficulty needs checking against evidence beyond the generator's own judgment.

Key takeaways · AI-distilled

The scoreboard in quadratic weighted kappa (agreement with the true label, credited partially for near misses): best LLM 0.578, ConvBERT 0.625. A small encoder trained on the task beats a frontier model prompted for it.
Both learned models also beat the hand-engineered feature baseline, so the story isn't neural versus classical. It's task-specific training versus general-purpose prompting.
Why prompting fails here: projecting the item down to a low-dimensional map, easy and hard items sit on top of each other. Difficulty barely registers in the text's meaning, so a model reading meaning has little signal to read.
No prompting configuration rescued it. Across several models and strategies the winner was plain zero-shot at temperature 0, meaning the ceiling was the approach, not the prompt.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.

Key quotes

“Zero-shot GPT-4.1 with a temperature of 0 yielded the highest item difficulty level prediction accuracy, with a quadratic weighted kappa (QWK) of 0.578.”

“However, LLMs' prediction accuracy was lower than that of ConvBERT (QWK = 0.625), which outperformed the best feature-based supervised machine learning model.”

“Further analysis showed that all LLMs struggled to label hard items; in particular, the current advanced GPT-5.4 tended to underestimate item difficulty levels.”

Read the source arxiv.org