← All IntelClip / AI AgentsStop patching everything in the prompt — fix the right layer
From From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI · ≈13:58
Calls out the industry anti-pattern of stuffing 'never do X' rules into prompts; with full-stack simulation you can route each fix to the harness, a skill, or structured output instead.
What’s in it
- Calls out the industry anti-pattern of stuffing 'never do X' rules into prompts; with full-stack simulation you can route each fix to the harness, a skill, or structured output instead.
Clip transcript
In this case, the process for improving agent becomes straightforward. You establish baseline in like on your benchmark, you run evaluation data set, you see the failures where it doesn't perform well, you change one thing, you rerun experiment and where you can use something like Arize to record your experiments. And then once you fix it, you rerun the full experimentation again. And then you can finally release to production. What it unlocks is to make sure that you fix issues correctly. There is a bit of an anti-pattern in the industry where like folks try to fix things in the prompt. And they populate the prompt with things like, "Never do this or only do that. Never output this critical that." Which is one way of handling it for sure, but with simulation, you control the full stack. You can evaluate the full stack. And you can make sure that fix lives in the correct place. You don't push everything to the prompt. You fix hardness if you want if you have context overload. Or you can if there is a missing procedure, you put it into skill. Or like if you need to have specific output schema, you put it as part of structured output.
Comments
Sign in to comment.
Loading comments…