← All IntelClip / AI AgentsVibe-based evaluation is the right first move
From How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads · ≈3:59
Argues counterintuitively that a non-scalable intuition pass beats a comprehensive eval on day one, because early prompt tweaks and architecture rewrites produce huge swings and a premature eval both slows and miscalibrates you.
What’s in it
- Argues counterintuitively that a non-scalable intuition pass beats a comprehensive eval on day one, because early prompt tweaks and architecture rewrites produce huge swings and a premature eval both slows and miscalibrates you.
Clip transcript
And uh so an interesting uh thing here that I think might be somewhat counterintuitive is that early on vibing can actually be kind of good for you. Uh and what I mean here by vibing is basically um doing things that are not exactly scalable to begin with. Um so when you're first uh starting out it may be that uh you you know you could uh take a track of basically just going ahead and making the super comprehensive eval right um but we found it actually works better to first do intuition based approach where you kind of um first see the capabilities and look at the outputs and at this stage it's pretty easy to tell what the issues actually are right uh so even though this is non-scalable it will still give you like a very good idea of when you change this what happens um and like uh it allows you to more quickly iterate as well. So at this stage prompt tweaks can also have like large performance gains you can make a radical change to the architecture um and your eval is not kind of like hindering you in this way. So it's like a very good way kind of you know like an early stage company of just like first you know doing something making more radical changes quickly. Um so yeah this way I think you can also get very familiar with what you're building what the failure patterns are and uh it gives you more of a a sense of depth and understanding it uh which allows you to hill climb in a targeted way and these will basically be very useful learnings when you're actually like building the more comprehensive EVO. Uh yeah, and then there's a chart here showing kind of uh you know if if you uh jump to scale to um these scaled raiders like too early uh it can cause you to kind of have like very big ups and downs as you might be iterating and calibrating the eval as you are struggling with uh changing the model radically.
Comments
Sign in to comment.
Loading comments…