Congrats to @Alibaba_Qwen on the release of Qwen3.8-Flash-Next, using the same architecture innovations as their upcoming Qwen4 model! Such innovations include:
🟠 51-billion-param N-gram Embedding to look up a table with very little extra computation, which means the embedding table can be offloaded to slower & less expensive tiers of DRAM
🟠 Gated Residual (GR): it seems like a lot of Chinese labs are now innovating on the res connections, like Kimi's AttentionRes and DeepSeek's mHC
🟠 Qwen Sparse Attention (QSA): lightning indexer to select context at micro-block granularity
Glad to see great Chinese open innovations along with end-to-end model weights to show these innovations can compose well together!
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
Why it matters
Three composable architecture choices — offloadable n-gram embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition →, gated residuals and micro-block sparse attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → — shipped together with open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition →, so the design can be inspected rather than inferred.
