An AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s shortened conversation history is useful only if it preserves what the agent needs next. Factorytested compression methods by asking follow-up questions about earlier facts, files, decisions and unfinished work. Its method merges new information into a persistent summary with dedicated sections.
In the company's model-judged evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →, Factory scored 3.70 out of 5 overall, compared with 3.44 for the tested Anthropic method and 3.35 for OpenAI's. These are scores on follow-up answers, not completed-task success rates. Tracking which files had been examined or changed remained weak across all three methods. The findings support evaluating retained working information alongside summary size.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
