Selecting The Right AI Evals Tool
hamel.dev- Category
- Other
- Type
- ARTICLE
- Builder
- @HamelHusain
- Added
- Jul 21, 2026
About
Over the past year, I’ve focused heavily on AI Evals , both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons. First, people focus too much on tools instead of the process, thinking the tool will be an off-the-shelf solution when it rarely is. Second, the tools change so quickly that comparisons become outdated immediately. Having used many of the popular eval tools, I can genuinely say t
What it can do
Assess AI evaluation tools using a panel of skilled data scientists
AI eval tools (e.g., Langsmith, Braintrust, Arize Phoenix) and a standardized homework assignment → Expert panel assessment and commentary on each tool's strengths and weaknesses
Compare eval vendors performing the same challenge
A common homework assignment completed by multiple vendors → Side-by-side observations of how each tool tackles an identical task
Provide recorded walkthroughs of eval tool usage
Vendor demonstrations with company representatives → Recorded process videos and live commentary
Define criteria for selecting an AI evals tool
Themes surfaced during expert review (workflow, developer experience, etc.) → A structured list of assessment criteria for teams to consider
Guide teams in matching a tool to their needs
A team's skillset, technical stack, and maturity level → Recommendations on which eval tool best fits the team's context
Evaluate developer experience and workflow friction
The process of going from observing a failure to iterating on a solution → Assessment of time-to-iterate and workflow smoothness (e.g., trace-to-playground)
Why it made the leaderboard
If you're choosing an evals tool for an AI product team, this walks through how expert data scientists actually assess Langsmith, Braintrust, and Arize Phoenix on the same task, focusing on failure-to-iteration workflow friction rather than feature checklists.
Media
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.