Designing Multi-User Agents for Group Chats and Wearables
Source
AI Engineer
Author
AI Engineer
Date
Why it matters
If you're building agents that sit in a group chat, shared workspace or wearable, this lays out two failure modes single-user designs don't surface: context bloat across many speakers, and cross-skill information leakage where two safe capabilities combine into a disclosure path.
The proposed per-user adapter guard model is a concrete alternative to regex-based redaction that misses injected or implied context.
A talk detailing eight months of building a personal AI agent for group chat and eventual glasses deployment, focused on two failure modes unique to multi-user settings: memory that must track long multi-person conversations without bloating context, and security risks where two individually safe skills can combine to leak information.
The speaker's mitigation involves an agent that audits its own skills to design a guard, plus a fine-tuned small model with per-user adapters that filters context-appropriate information and detects prompt injection that regex-based filters miss.
Transcript
Almost every agent today is built for one user, and Sai Krishna Rallabandi has spent about eight months on what breaks when you drop one into a group chat instead. Running a personal agent across a real group, and eventually onto glasses, forces two hard problems. The first is memory: conversations that last for days across several people, where the agent has to track who said what and surface only what is relevant without bloating its context. The second is security, and it is where the talk digs in. He points to a finding that two skills, each safe on its own, can collide once they share infrastructure, so a reporting agent leaks information at low frequency. His defense is to have an agent read everything and design a guard, then fine tune a small model with a per user adapter that only releases information appropriate to the context and catches prompt injection that regex would miss. The through line is that group and wearable settings break the single user assumptions most agent harnesses are built on, so memory and security have to be redesigned around the group. Speaker info: - https://x.com/Saikallis9012 - https://www.linkedin.com/in/sai-krishna-rallabandi-8595418b/ - https://saikrishnarallabandi.github.io/ Timestamps: 0:00 - Introduction: agents for groups, not one user 1:28 - Why single user agents fall short 2:45 - Eight months in a real group chat 3:47 - What changes when the agent joins a group 5:17 - Two problems: guarding and memory 5:55 - Securing an agentic system 7:44 - When two safe skills collide 9:27 - Designing a guard agent 10:21 - A small model with per user adapters 11:49 - Catching prompt injection 12:29 - Designing memory for groups 15:29 - Context growth and token cost
