Factory, which also makes one of the three methods it compares, evaluates compressed AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → conversations by asking what the agent can still remember: the original problem, changed files, past decisions and next steps. Its evaluation report compares three compression methods using questions about the discarded history, with another LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → grading the answers.
The weakest area across all three methods was tracking files and changes, scoring between 2.19 and 2.45 out of five. Factory's own structured summaries also struggled here. This makes file-state preservation a concrete concern for long coding sessions: a compact summary may remain readable while losing information needed to resume work accurately.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.
