← All IntelClip / AI AgentsLong-session evals: load 10 turns, test the 11th
From How we solved Context Management in Agents — Sally-Ann Delucia · ≈7:58
Users don't restart chats, so context failures surface late; replaying a deep conversation state turns those late-session bugs into a repeatable test instead of a user report.
What’s in it
- Users don't restart chats, so context failures surface late; replaying a deep conversation state turns those late-session bugs into a repeatable test instead of a user report.
Clip transcript
working quite well for us. But we had another problem as we were kind of deciding how to to handle context management, which is long sessions and I think that this is something that a lot of people run into, which is users don't usually restart their chats. Um you know, I I think there are different approaches to this. Some people I know like if you're using Claude or Cursor, you know, you pull everything in one chat. Some people like to have other ones, but we really learned with Alex everybody kind of wants to stay in one chat as they're traveling to pages to pages. So, our conversations grow and our failures appear late. So, uh when we first did this smart truncation, it seemed like it was working, um but then as we saw these longer and longer conversations, there were failures happening and we didn't know about them too late until like a user reported it or I was looking at the data and I realized that Alex kind of started to forget things way late into the conversation. And so, the solution that we came up here is long session evals and I want to include this because, you know, it's maybe not um related exactly to how you handle context management, but it's a really helpful signal in understanding how your context management is doing. Uh because I think long sessions are something that naturally happen with these applications. Um and so, what we end up doing is we load 10 turns and then we test the 11th to understand how the context is doing. And so, these bugs really become testable. I don't have to wait till I find it or a user reports it. So, uh wanted to share a little bit
Comments
Sign in to comment.
Loading comments…