← All IntelClip / AI AgentsA continually-adapting relevance scorer, and KV-cache-aware injection
From Designing Multi-User Agents for Group Chats and Wearables · ≈14:54
Two implementation details for memory serving: knowledge-based compaction via a scorer that re-ranks as group context evolves, and an injection engine that respects KV caching so memory edits don't invalidate the cache.
What’s in it
- Two implementation details for memory serving: knowledge-based compaction via a scorer that re-ranks as group context evolves, and an injection engine that respects KV caching so memory edits don't invalidate the cache.
Clip transcript
organizing information in form of a graph etc. So uh the other interesting aspect when it comes to memory would be what uh what we store and what we forget. As the memory becomes bloated and bloated, we might want to train a very simple machine learning model or an SLM to try and figure out what are the important concept context uh information that we need to store. And a typical approach, a a very simple approach discussed in the paper learning what not to forget is where they learn a relevance scorer which keeps scoring continually. The important word here being continually because in a group chat setting or in a group work setting the context keeps evolving and different things become important at different points and we can once we have a relevant scorer which is continuously adapting we can save much more on the tokens because now we are doing knowledge based compaction. One other thing when it comes to serving these stored memories is uh whether whether we use cloud cloud-based serving or if we use local based serving we need to be aware of the fact that we all of us use KV caches and KV caches break when we try to do something cute with respect to the model. Therefore building an injection engine which is a aware of KV caching also helps there.
Comments
Checking sign-in…
Loading comments…