← All IntelClip / Other
Skepticism of low-level prompt tuning vs. model-driven self-correction
From The State of Model Routing — NVIDIA, Cognition, OpenRouter · ≈42:15
“I'm actually personally less bullish on these kind of like low level mechanical prompt tuning harnesses versus just telling like a smart model like here is the decision that was made and the context figure out why it went wrong.”
“Sometimes you can do something as dumb as asking the model why did you do this instead of this and cite the prompts and then just have your agent your dev and just go and update the prompts, rerun the the test as a regression, make sure make sure it changes.”
“Like it's a lot heavier weight of a system, but I kind of trust the intelligence of a system like that a lot more.”
What’s in it
- Argues against low-level gradient-based prompt tuning frameworks
- Shows a simpler technique: ask the model why it erred, then fix prompts
- Explains why trusting a smart model's self-diagnosis beats mechanical tuning harnesses
Clip transcript
something that we're we're thinking about. >> Have you guys looked into prompt tuning and do you find it useful like say Japa? >> Yeah, so there are like these prompt tuning frameworks from like a few years ago that tried to do some kind of like gradient descent type thing. I'm like I'm actually personally less bullish on these kind of like low level mechanical prompt tuning harnesses versus just telling like a smart model like here is the decision that was made and the context figure out why it went wrong. Sometimes you can do something as dumb as asking the model why did you do this instead of this and cite the prompts and then just have your agent your dev and just go and update the prompts, rerun the the test as a regression, make sure make sure it changes. Like it's a lot heavier weight of a system, but I kind of trust the intelligence of a system like that a lot more.
Recommended reads
Comments
Sign in to comment.
Loading comments…