llama.cpp v0.4.0
- Source
- github.com
- Author
- github-actions[bot]
- Date
Context
The model-running library llama.cpp adds a way to reduce memory occupied by certain large lookup tables. Version 0.4.0 includes on-demand tensor reading: eligible numerical tables can be read from the model file as needed instead of being loaded fully into RAM.
The implementation discussion explains the trade-off. Only tensors explicitly marked as eligible, such as per-layer tables, use it. In the PR's Gemma 4 E4B test, peak memory fell from 7.37 GB to 6.16 GB while generation slowed by roughly 8 to 11 percent, and the author says the cost is larger for small models because reads are slow relative to generation. It is a targeted memory-saving option, with speed depending on the model and loading configuration.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Checking sign-in…
Loading comments…