• Sliding-window attention lets each token look only at the last ~4,096 tokens instead of a 256k context, cutting per-layer cost from O(n squared) to roughly O(n times window). Arcee's Trinity runs 3 local layers per global one.
• Trinity multiplies the attention output by a learned elementwise gate before the final projection. That damps attention sinks — positions that soak up attention for no reason — and improved both long-sequence behavior and training stability.
• Trinity initializes the second RMSNorm gain in each block at about 1/sqrt(L), where L is the layer count. Residual updates start tiny and grow as training finds the right scale — a stability trick hiding inside a Gemma 3-looking norm layout.
• Kimi K2.5's ablation: for a fixed budget of image tokens, mixing a smaller number in early during pretraining beats dumping more in later. When the model first sees vision matters more than how much of it there is.
• Multi-token prediction bolts small extra heads onto each position to predict tokens t+1 through t+4 and sums the losses. The extra signal speeds training while inference still emits one token at a time; Step 3.5 Flash hits 100 tokens/sec at 128k context.

• open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
• mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
• context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
• pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.

A technical walkthrough of ten open-weight LLM architectures — what actually changed in MoE routing, attention and multi-token prediction.

• articleBeyond Standard LLMs
• articleFrom GPT-2 to gpt-oss: Analyzing the Architectural Advances
• articleUnderstanding the 4 Main Approaches to LLM Evaluation (From Scratch)
• articleUnderstanding and Implementing Qwen3 From Scratch

• articleThe Big LLM Architecture ComparisonSebastian Raschka, PhD
• articleBeyond Standard LLMsSebastian Raschka, PhD
• articleRecent Developments in LLM Architectures: KV Sharing, mHC, and Compressed AttentionSebastian Raschka, PhD

Checking sign-in…

Loading comments…