← All IntelClip / AI Agents
Agent-generated skill improvements need a human review gate
From Skill issue: Lessons from skilling up coding agents to use Langfuse - Marc Klingen, Clickhouse · ≈20:09
The workable division of labor for self-improving skills: let the optimizer be out-of-the-loop idea generation, keep humans reviewing every suggested change.
What’s in it
- The workable division of labor for self-improving skills: let the optimizer be out-of-the-loop idea generation, keep humans reviewing every suggested change.
Clip transcript
I DON'T KNOW DO WE HAVE TIME FOR A QUESTION? OKAY. YEP. UH YEAH, it was kind of um like I mean you kind of want to be out of the loop for the experimentation and then just review the suggested changes. Uh so it was kind of um like experimental things uh give us like all sorts of different recommendations and then human review the suggestions. Like we didn't accept all because many didn't make sense because our target function wasn't perfect. It was really difficult to get to like a very perfect target function, but it's good at just creating ideas and then uh we human reviewed all of the ideas to make the changes to the skill.
Recommended reads
- clipHuman PR review plus automated skill checksAI Engineer
- clipSpend human experts only where agent and verifiers disagreeAI Engineer
articleRethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple RoundsYuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, Yangqiu Song
Comments
Sign in to comment.
Loading comments…