Your KV Cache Can Spot Attacks on Your AI Agents - Ari Azarafrooz, Intrinsec AI

Offers a defense layer that works where text filters cannot, by flagging unusual KV pages against a baseline learned from normal traffic.

• Ari Azarafrooz of Intrinsec AI proposes learning a baseline of KV cache pages from normal traffic and flagging pages that look unusual, instead of guarding inputs and outputs with judge models or a separate classifier for every failure.
• Because the signal comes from the inference engine's own KV cache rather than from text, it can catch a hidden spoken instruction in a voicemail, a case with no text to filter.
• He covers where the approach plugs into vLLM, SGLang, LMCache, llm-d and NVIDIA Dynamo, and names two hard production problems: models that keep changing, and KV data too large to move in real time.

• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

• videoWill AI Agents Pretend to Be Human to Get Past a Check? - Maty Bohacek, Stanford
• videoThe One Thing Every AI Agent Framework Agrees On - Mislav Ivanda, Daytona
• videoAI Writes Code Faster Than Uber Can Merge It - Preetam Dwivedi & Manjari Akella, Uber
• videoCan AI Agents Beat Wall Street at Predicting Earnings? - Ashwin Paranjape, Samaya AI

• videoWhat Is an Inference Engine, Anyway? — Charles Frye, ModalAI Engineer
• articleUnderstanding and Coding the KV Cache in LLMs from ScratchSebastian Raschka, PhD
• videoWhat's New in Inference Engineering — Philip Kiely, BasetenAI Engineer

Checking sign-in…

Loading comments…