Evals for taste: Hill-climbing a slide-generation agent
- Source
- youtube.com
- Author
- Claude
- Date
Context
A presenter in a workshop video from the Claude channel uses a slide-generating agent to show how evals, systematic tests with scoring rules, turn a vague sense that output got worse into something to act on. Code graders, such as counting emojis or slides, are fast and exact but brittle. Model graders score things like layout against a rubric, and are flexible but cost more, vary between runs and need . Human graders are the most nuanced and most expensive.
The demo scored a first deck, found emojis, small fonts and clutter, then revised the , required diagrams and added a QA loop that hunts for flaws, re-scoring each time. Judge scores rose to about 4.2 to 4.4 out of five. But a stronger model with only the basic also scored high, including 5 for images on a deck with none. The presenter takes that as a sign the judges were poorly anchored, and advises asking for reasons before a score, because a model that states a number first tends to defend it.
- calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
- system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
- Prompt — Information or instructions supplied to a model; in a chat, the request you type is often called your prompt.
- Session — One conversation or piece of work as a harness keeps it: a record of messages, tool calls and observations that can be saved and resumed.
Checking sign-in…
Loading comments…





