1,000 Agent Tasks in a Sandbox: What Breaks When LLMs Run Code — Kevin Orellana, AWS
Source
youtube.com
Author
AI Engineer
Date
Context
An AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s failed task can expose a broken tool interface or an execution limit. In an Amazon engineer's account of testing AgentCore, a coding agent generated and ran workflows against a cloud code sandboxAn isolated environment where AI-generated code or agent actions run without being able to touch anything real.Full definition → and browser. An image upload-and-download test revealed an encoding error in the tool interface.
Other failures arose inside the execution environment: browser tests exhausted memory, and written files could not be retrieved. Separating those categories changes the diagnosis. Repairing how a tool passes data addresses a different problem from changing the resources or file access available to the workload.
Key takeaways · AI-distilled
Orellana wrapped AgentCore's code sandbox and cloud browser in an MCPThe Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.Full definition → server, had a coding agent generate 1,000 tests across 10 domains, and logged results to SQLite, arguing results belong in a database rather than markdown files.
Failures surfaced included image encoding bugs, null bytes, unreadable files and browser memory exhaustion.
He separates interface-layer bugs from sandbox-layer limits such as memory, file permissions, seccomp and eBPF, and argues those sandbox limits decide whether an agent can finish its task. He also covers the limits of LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition →-generated tests.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
videoHow Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs
videoAuth for Agents: Let AI Sign Up for Your Product with auth.md — Michael Grinich, WorkOS
videoDecoupling the Brain from the Hands: Claude Managed Agents (Part 1 of 4) — Anthropic
videoThe Next Trillion Internet Users Still Don’t Have an Identity — Adi Singh, AgentMail
