context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
If you're hitting degraded quality on long-document tasks, this shows a planner/worker/manager decomposition that lets cheaper open models like Llama-3-70B and Qwen-72B beat single-shot GPT-4o — a practical way to cut cost while improving reliability on large contexts.
Key quotes
“The Intuition: Don't ask one genius to read a library in an hour. Ask ten interns to read one book each.”
Together AI
“Model Noise (The "Brain Fog"): Models don't just forget linearly; they get overwhelmed. Our research shows that model confusion grows superlinearly with input length. It is mathematically better to split the task because you reset that confusion counter with every new chunk in a shorter length”
Together AI
“If you naively ask workers to "Find the 2nd smallest number" in their specific chunk, the final manager will fail because the global 2nd smallest number might be the smallest number in a specific chunk.”
Together AI
“We found that testing just 5 random samples is sufficient to find the optimal chunk size due to the predictability of the noise curve.”
Together AI
“If your task requires tracking a subtle clue from Page 1 that connects to Page 100 (like the "Dialogue Character Inference" task in our paper), the "Divide" step breaks the necessary context.”
