Vibeleaderboard
← All Intel
Intel / repo

Ollama served my 40k-context model at 4k, silently

Source
github.com
Author
satoshiakiyama
Date

Context

A model's advertised capacity can differ from the amount its running server actually uses. In a report using Ollama 0.32.9 on Windows, the author found a Qwen model declaring 40,960 running with a 4,096-token window. A long test prompt received a successful response even though the server reported evaluating only 2,050 prompt tokens.

Setting the request's context window explicitly to 16,384 increased the evaluated input to 7,855 tokens and recovered a value placed at the beginning of the prompt. This makes the server's active window and evaluated-input count consequential when diagnosing apparent forgetting: the model may never have received the missing information. The finding is specific to the reported setup.

Terms in this piece · Glossary
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Recommended reads
Comments

Checking sign-in…

Loading comments…