Sebastian Raschka

Start here

A book companion hub for implementing and training a GPT-style language model with PyTorch.

Follow the companion hub’s read, watch, code and exercise sequence. Python experience and basic neural-network familiarity help; the implementation starts below the level of model APIs.

Sebastian Raschka releases an 11-hour, eight-part video course that accompanies his Build Reasoning Model From Scratch book. It starts from a pre-trained LLM and implements evaluation, inference scaling, reinforcement learning, and distillation in Python and PyTorch.

Sebastian Raschka traces language-model classification from bag-of-words to the Jev model, guesses at its methodology, and compares it with LLMs and special-purpose classifiers on speed, cost and generality.

Sebastian Raschka reviews GPT-6 Astra's capabilities, then explains looped transformer / recurrent-depth architecture in depth and whether it accounts for OpenAI hiding the model's chain-of-thought, drawing on recent research.

A recorded lecture with transcript that walks slide by slide through the mechanism behind Anthropic's text watermarking of Claude output, expanded from a short social post into roughly 50 slides and 48 minutes of detail.

Sebastian Raschka builds a small AI-text detector from scratch, then turns it into a verifier used to train a small language model to write text that evades detection, as a study of where such detectors break down.

An in-depth technical article explaining how large language models learn to support low-, medium-, and high-effort reasoning modes. It covers reasoning model training via RLVR, think tokens, on/off reasoning switches, and how reasoning effort settings work in models like GPT-5, Qwen3, and gpt-oss.

A tutorial for assembling a fully local coding agent from open-source tools and open-weight models, pairing an agent harness with a model served by a local inference runtime.

A curated, categorized reading list of notable LLM research papers published from January to May 2026, organized into topics like architecture, reasoning, reinforcement learning, and agent systems. It's a reference resource for researchers and practitioners who want to track the field's developments.

An in-depth technical article analyzing recent open-weight LLM architecture innovations focused on long-context efficiency, covering KV sharing and per-layer embeddings in Gemma 4, attention budgeting in Laguna XS.2, compressed convolutional attention in ZAYA1-8B, and DeepSeek V4's mHC design. Useful for ML engineers and researchers who want to understand how modern transformers reduce KV cache size and attention costs.

A learning-oriented walkthrough of how to reverse-engineer and understand new open-weight LLM architectures by inspecting Hugging Face config files and reference implementations rather than relying on sparse technical papers. Useful for ML practitioners who want to deeply understand how modern model architectures actually work.

A minimal but fully working coding agent implemented in pure Python, designed to demonstrate the six core building blocks of coding agents like Claude Code and Codex CLI. It serves as an educational reference for understanding how agent harnesses use tools, memory, and repo context to make LLMs more capable at software tasks.

An educational article and visual gallery explaining the evolution of attention mechanisms in modern LLMs, from Multi-Head Attention (MHA) and Grouped-Query Attention (GQA) to Multi-Head Latent Attention (MLA) and sparse/hybrid architectures. Each concept is illustrated with visual model cards and real-world example architectures.

A technical roundup comparing ten open-weight LLM architectures released in early 2026, including Arcee Trinity, Kimi K2.5, Step 3.5 Flash, Qwen3-Coder-Next, and GLM-5. It breaks down architectural details like Mixture-of-Experts, sliding window attention, gated attention, and multi-token prediction for each model.

An in-depth educational article that categorizes and explains inference-time scaling techniques for improving LLM reasoning, covering methods like chain-of-thought prompting, self-consistency, best-of-N ranking, rejection sampling, self-refinement, and search over solution paths. It's a companion piece to a book chapter on building reasoning models from scratch.

An in-depth year-in-review article covering the major developments in large language models throughout 2025, including DeepSeek R1, RLVR, GRPO, inference-time scaling, and architectural trends, plus predictions for 2026 and 2027. Written for practitioners and researchers who want a technically grounded synthesis of the year's LLM progress.

An in-depth technical article walking through the architectural evolution of DeepSeek's open-weight LLMs from V3 to V3.2, explaining concepts like Multi-Head Latent Attention, RLVR reasoning training, and the new DeepSeek Sparse Attention mechanism.

An in-depth educational article by Sebastian Raschka exploring alternative LLM architectures beyond standard autoregressive transformers, including linear attention hybrids, text diffusion models, code world models, and small recursive transformers. It breaks down architectures like Qwen3-Next, Kimi Linear, MiniMax-M1/M2, and DeepSeek V3.2 with technical explanations and code illustrations.

An educational article that explains the four primary methods for evaluating large language models — multiple-choice benchmarks, verifiers, leaderboards, and LLM judges — with from-scratch PyTorch code examples. It helps readers build a mental map for interpreting benchmarks, leaderboards, and research papers.

An in-depth educational article that walks through the Qwen3 LLM architecture and reimplements it from scratch in pure PyTorch. It covers both the dense and Mixture-of-Experts variants, helping readers understand how modern open-weight models work under the hood.

An in-depth technical article breaking down the architectural design choices in OpenAI's gpt-oss-120b and gpt-oss-20b open-weight models, comparing them against GPT-2 and Qwen3. It walks through MoE, RoPE, GQA, sliding-window attention, RMSNorm, MXFP4 optimization, and more.

An in-depth technical article that compares the architectural designs of modern flagship open-weight LLMs, from DeepSeek-V3 and OLMo 2 to Kimi K2 and Gemma. It breaks down design choices like Multi-Head Latent Attention, Mixture-of-Experts, normalization placement, and QK-Norm.

A topic-organized collection of 200+ large language model research papers from the first half of 2025, curated by Sebastian Raschka. Papers are grouped by themes like reasoning models, reinforcement learning, and multimodal models for easy reference.

A hands-on technical tutorial that explains how key-value (KV) caches speed up LLM inference, walking through a from-scratch, human-readable PyTorch implementation. It compares text generation with and without caching and demonstrates a ~5x speedup on a 124M parameter model.

A ~15-hour video course by Sebastian Raschka that teaches you how to build a large language model from scratch in Python — covering tokenization, attention mechanisms, pretraining, and finetuning. It serves as standalone companion content to his 'Build a Large Language Model (From Scratch)' book.

LLMs from scratchgithub.com

Step-by-step notebooks implementing a language model and its training stages.

Reasoning from scratchgithub.com

Code for building a reasoning language model in PyTorch.