Clip / Other

Human-in-the-Loop Confidence Checks and Token-Limit Stop Reasons

From Anthropic's CCA Exam as a Field-Guide for Agentic Engineering — Frank Coyle, UC Berkeley · ≈10:35

“We're not running a tool anymore.”

“Now, then we take the answer, and this is an opportunity for you to have a human in the loop potentially.”

“You check the confidence.”

“If you don't, then you escalate to a human.”

What’s in it

Explains why checking an LLM's stop reason matters
Shows where to insert human review after a tool loop ends
Flags token-limit cutoffs as a silent failure mode

Watch this moment ↗

Clip transcript

We're not running a tool anymore. We're end the end of our loop. Bingo. Now, then we take the answer, and this is an opportunity for you to have a human in the loop potentially. You check the confidence. If it looks good, you keep it. If you don't, then you escalate to a human. So, now there's another reason why you need to make sure you check your stop reason. One of the stop reasons may be you have run out of tokens, and this response is based on partial when the LLM had to stop. And it's going to give you a response, but if you have run out of tokens, then you need to take action.

Intel