← All IntelClip / AI AgentsModel upgrades silently break skills
From Skills as the Interface to Agentic Products · ≈11:42
“skills without evals are really just wishful thinking.”
“Skills are not the documentation and a lot of people treat them like that and skills are really the contracts versioned to a model.”
What’s in it
- Explains why a model upgrade broke previously working AI 'skills'
- Reveals new models over-attend to the start of prompts, ignoring end instructions
- Argues skills are versioned contracts with a model, not documentation
Clip transcript
So we updated our uh stack to a new model and our agent start uh failing because it was not obeying the skills. Nothing was changed. Not a single line in the skill was changed but still it failed. And when when we dig under the hood what was happening it was that this like new model was very focusing on beginning of the skill and we had very critical instruction at the end of the skill. So that's why it's very important to run evals and skills without evals are really just wishful thinking. Skills are not the documentation and a lot of people treat them like that and skills are really the contracts versioned to a model. So whenever you're upgrading a model, make sure to rerun uh your eval.
Recommended reads
- clipSimulation-awareness undermines behavioral evalsAI Engineer
- clipHow the eval target has changed since 2023AI Engineer
articleRethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple RoundsYuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, Yangqiu Song
Comments
Sign in to comment.
Loading comments…