Llmcrit Teaching Large Language Models To Use Criteria 2024 03 02

LLMCRIT is a framework that lets a language model give written feedback against explicit criteria rather than general impressions. The researchers derive criteria semi-automatically from existing human-written guidelines, with a model extracting them and manual review refining them, then attach example demonstrations to each criterion. They tested it on three tasks: paper introductions, Python code, and Reddit posts.

The paper reports that supplying criteria produced feedback with more critiques and suggestions. Demonstrations could make the critiques more helpful, but sometimes distracted the model into commenting on the example instead of the submitted text. The evaluation was primarily model-based, so the findings support a particular feedback workflow rather than a general guarantee of better editing.