Intel / article

Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading

Source: newsletter.semianalysis.com
Author: Bryan Shan
Date: Published Sep 18, 2026

From the Daily Brief

SemiAnalysis published a technical breakdown of Engram, a model architecture that extends token embeddings with learned multi-token lookups addressable by token ID. Because those extra rows are addressable ahead of time, they can prefetch from host DRAM or even SSD instead of sitting permanently in a GPU's high-bandwidth memory, which is the scarcest and most expensive resource in a serving stack. The pitch is that a model gets a larger effective embedding table, and the retrieval-style capacity that comes with it, without a proportional increase in HBM footprint, since the codesign moves the lookup off the critical memory path rather than compressing it. This is a supply-side argument rather than a benchmark claim: it targets the same HBM bottleneck that has been driving hardware vendors toward shorter HBM stacks and inference providers toward memory-tiering tricks, but as an architecture change made at training time rather than a serving-time patch. Teams that are HBM-constrained rather than compute-constrained, which is increasingly the common case for large-context serving, are the ones for whom this design tradeoff will matter most directly.

Read the 2026-09-19 Brief →

Context

Moving a model's stored information to cheaper memory does not automatically make serving it cheaper. SemiAnalysis examines Engram, a mechanism that retrieves learned vectors for recurring text patterns. Because each step needs only a small selection from a large table, the table can reside outside the accelerator's high-bandwidth memory. Its report compares serving paths using system RAM and SSD-backed files.

In the report's SSD implementation, fetching those rows adds CPU coordination, collection and transfers back to the accelerator. Even files already cached in RAM retain that overhead. On the B200 configurations SemiAnalysis measured, the RAM-based setup delivered better responsiveness and more output per dollar. That finding concerns this unoptimized SSD implementation: SemiAnalysis says SSD storage left the same expensive accelerators in place, so reclaimed RAM did not by itself yield a cheaper server.

Key takeaways · AI-distilled

Engram -row addresses depend only on ID, not hidden state, so the runtime can prefetch rows from host DRAM while earlier layers compute, avoiding the need to move whole weight matrices the way ordinary parameter offloading does.
DeepSeek-V4.1-Flash's Engram table runs about 189 GiB; SemiAnalysis memory-mapped it to SSD and found DRAM-offloading the table can beat keeping it in HBM across most of the pareto frontier, even on GPU SKUs with high HBM capacity.
Probing DeepSeek-V4.1-Flash's Engram gate scores shows it mostly retrieves names, code fragments, relational phrasing, and boilerplate, evidence the learned memory optimizes the training objective rather than deliberately storing 'important' facts.
SemiAnalysis replicated DeepSeek's original two unreleased Engram models on fineweb-edu at an estimated 6E18 FLOPs per run and reproduced the same U-shaped scaling curve and earlier-layer representations resembling later-layer ones.
Engram lowers HBM needs per unit of quality but is framed as architecture adapting to constraints, not an escape from HBM demand, arriving as NVIDIA cuts planned Rubin Ultra HBM from 1024GB to about 200GB per chip.

Terms in this piece · Glossary

token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.

Read the source newsletter.semianalysis.com