← All IntelClip / AI AgentsWhere evals fit: online evals layered onto production traces
From From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize · ≈18:04
Explains the plumbing — eval results live as data on the trace itself, so the same skills that pull traces also give the agent aggregate eval signal.
What’s in it
- Explains the plumbing — eval results live as data on the trace itself, so the same skills that pull traces also give the agent aggregate eval signal.
Clip transcript
Anyone else? >> Oh yeah. Okay. One more. Thanks for the talk. Um, there was quite a few mention of eval but you know I'm looking at the traces so you know I understand the concept of traces but where where did the evals come in when you have that signal that says hey something broke in production. >> Yeah. So so the so the eval typically will the um the eval essentially are running and being layered on typically to the production traces something we call online evals. Um let me see if this one has an example here of it. Um so so eval actually are data on the trace itself and so the agent knows how to uh grab the data from traces knows how to um visualize and you know the skills to basically pull data for for the aggregate values of evals across the traces so that so the the skills that you give uh the harness allow it to get the data on the eval from from the traces. Um so eval are kind of like I view them as le at least a first generation eval evals which are elements a judge um as a as a AI layer that
Comments
Sign in to comment.
Loading comments…