Inside Qwen3.8-Flash-Next: n-gram embeddings, gated residuals, sparse attention
- Source
- x.com
- Date
Congrats to @Alibaba_Qwen on the release of Qwen3.8-Flash-Next, using the same architecture innovations as their upcoming Qwen4 model! Such innovations include: 🟠 51-billion-param N-gram Embedding to look up a table with very little extra computation, which means the embedding table can be offloaded to slower & less expensive tiers of DRAM 🟠 Gated Residual (GR): it seems like a lot of Chinese labs are now innovating on the res connections, like Kimi's AttentionRes and DeepSeek's mHC 🟠 Qwen Sparse Attention (QSA): lightning indexer to select context at micro-block granularity Glad to see great Chinese open innovations along with end-to-end model weights to show these innovations can compose well together!
Three composable architecture choices — offloadable n-gram , gated residuals and micro-block sparse — shipped together with , so the design can be inspected rather than inferred.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
Checking sign-in…
Loading comments…