Intel
A super long overdue (3+ years?) post on scaling laws. Compute is expensive. Sca
A super long overdue (3+ years?) post on scaling laws. Compute is expensive. Scaling laws are a way to help us reason about the optimal compute allocation between data and model size before committing to a large run. The post covers what scaling laws predict, how
postlilianweng
From GAN to WGAN
[Updated on 2018-09-30: thanks to Yoonju, we have this post translated in Korean !] [Updated on 2019-04-18: this post is also available on arXiv .] Generative adversarial network (GAN) has shown great results in many generative tasks to replicate the real-world rich content such as images, human language, and music. It is inspired by game theory: two models, a generator and a critic, are competing with each other while making each other stronger at the same time. However, it is rather challengin
articlelilianweng.github.io
Anatomize Deep Learning with Information Theory
Professor Naftali Tishby passed away in 2021. Hope the post can introduce his cool idea of information bottleneck to more people. Recently I watched the talk “Information Theory in Deep Learning” by Prof Naftali Tishby and found it very interesting. He presented how to apply the information theory to study the growth and transformation of deep neural networks during training. Using the Information Bottleneck (IB) method, he proposed a new learning bound for deep neural networks (DNN)
articlelilianweng.github.io
Learning Word Embedding
Human vocabulary comes in free text. In order to make a machine learning model understand and process the natural language, we need to transform the free-text words into numeric values. One of the simplest transformation approaches is to do a one-hot encoding in which each distinct word stands for one dimension of the resulting vector and a binary value indicates whether the word presents (1) or not (0).
articlelilianweng.github.io
The Multi-Armed Bandit Problem and Its Solutions
The algorithms are implemented for Bernoulli bandit in lilianweng/multi-armed-bandit . Exploitation vs Exploration The exploration vs exploitation dilemma exists in many aspects of our life. Say, your favorite restaurant is right around the corner. If you go there every day, you would be confident of what you will get, but miss the chances of discovering an even better option. If you try new places all the time, very likely you are gonna have to eat unpleasant food from time to time. Similarly,
articlelilianweng.github.io
Policy Gradient Algorithms
[Updated on 2018-06-30: add two new policy gradient methods, SAC and D4PG .] [Updated on 2018-09-30: add a new policy gradient method, TD3 .] [Updated on 2019-02-09: add SAC with automatically adjusted temperature ]. [Updated on 2019-06-26: Thanks to Chanseok, we have a version of this post in Korean ]. [Updated on 2019-09-12: add a new policy gradient method SVPG .] [Updated on 2019-12-22: add a new policy gradient method IMPALA .] [Updated on 2020-10-15: add a new policy gradient method PPG &
articlelilianweng.github.io
Implementing Deep Reinforcement Learning Models with Tensorflow + OpenAI Gym
The full implementation is available in lilianweng/deep-reinforcement-learning-gym In the previous two posts, I have introduced the algorithms of many deep reinforcement learning models. Now it is the time to get our hands dirty and practice how to implement the models in the wild. The implementation is gonna be built in Tensorflow and OpenAI gym environment. The full version of the code in this tutorial is available in [lilian/deep-reinforcement-learning-gym] .
articlelilianweng.github.io
Attention? Attention!
[Updated on 2018-10-28: Add Pointer Network and the link to my implementation of Transformer.] [Updated on 2018-11-06: Add a link to the implementation of Transformer model.] [Updated on 2018-11-18: Add Neural Turing Machines .] [Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the show-attention-tell paper; moved it to Self-Attention section.] [Updated on 2020-04-07: A follow-up post on improved Transformer models is here .]
articlelilianweng.github.io
From Autoencoder to Beta-VAE
[Updated on 2019-07-18: add a section on VQ-VAE & VQ-VAE-2 .] [Updated on 2019-07-26: add a section on TD-VAE .] Autocoder is invented to reconstruct high-dimensional data using a neural network model with a narrow bottleneck layer in the middle (oops, this is probably not true for Variational Autoencoder , and we will investigate it in details in later sections). A nice byproduct is dimension reduction: the bottleneck layer captures a compressed latent encoding. Such a low-dimensional represent
articlelilianweng.github.io
Flow-based Deep Generative Models
So far, I’ve written about two types of generative models, GAN and VAE . Neither of them explicitly learns the probability density function of real data, $p(\mathbf{x})$ (where $\mathbf{x} \in \mathcal{D}$) — because it is really hard! Taking the generative model with latent variables as an example, $p(\mathbf{x}) = \int p(\mathbf{x}\vert\mathbf{z})p(\mathbf{z})d\mathbf{z}$ can hardly be calculated as it is intractable to go through all possible values of the latent code $\mathbf{z}$
articlelilianweng.github.io
Meta-Learning: Learning to Learn Fast
[Updated on 2019-10-01: thanks to Tianhao, we have this post translated in Chinese !]
articlelilianweng.github.io
Are Deep Neural Networks Dramatically Overfitted?
[Updated on 2019-05-27: add the section on Lottery Ticket Hypothesis.] If you are like me, entering into the field of deep learning with experience in traditional machine learning, you may often ponder over this question: Since a typical deep neural network has so many parameters and training error can easily be perfect, it should surely suffer from substantial overfitting. How could it be ever generalized to out-of-sample data points?
articlelilianweng.github.io
A (Long) Peek into Reinforcement Learning
[Updated on 2020-09-03: Updated the algorithm of SARSA and Q-learning so that the difference is more pronounced. [Updated on 2021-09-19: Thanks to 爱吃猫的鱼, we have this post in Chinese ].
articlelilianweng.github.io
Domain Randomization for Sim2Real Transfer
In Robotics, one of the hardest problems is how to make your model transfer to the real world. Due to the sample inefficiency of deep RL algorithms and the cost of data collection on real robots, we often need to train models in a simulator which theoretically provides an infinite amount of data. However, the reality gap between the simulator and the physical world often leads to failure when working with physical robots. The gap is triggered by an inconsistency between physical parameters (i.e.
articlelilianweng.github.io
Meta Reinforcement Learning
In my earlier post on meta-learning , the problem is mainly defined in the context of few-shot classification. Here I would like to explore more into cases when we try to “meta-learn” Reinforcement Learning (RL) tasks by developing an agent that can solve unseen tasks fast and efficiently.
articlelilianweng.github.io
Evolution Strategies
Stochastic gradient descent is a universal choice for optimizing deep learning models. However, it is not the only option. With black-box optimization algorithms, you can evaluate a target function $f(x): \mathbb{R}^n \to \mathbb{R}$, even when you don’t know the precise analytic form of $f(x)$ and thus cannot compute gradients or the Hessian matrix. Examples of black-box optimization methods include Simulated Annealing , Hill Climbing and Nelder-Mead method .
articlelilianweng.github.io
Self-Supervised Representation Learning
[Updated on 2020-01-09: add a new section on Contrastive Predictive Coding ]. [Updated on 2020-04-13: add a “Momentum Contrast” section on MoCo, SimCLR and CURL.] [Updated on 2020-07-08: add a “Bisimulation” section on DeepMDP and DBC.] [Updated on 2020-09-12: add MoCo V2 and BYOL in the “Momentum Contrast” section.] [Updated on 2021-05-31: remove section on “Momentum Contrast” and add a pointer to a full post on “Contrastive Representation L
articlelilianweng.github.io
Curriculum for Reinforcement Learning
[Updated on 2020-02-03: mentioning PCG in the “Task-Specific Curriculum” section. [Updated on 2020-02-04: Add a new “curriculum through distillation” section.
articlelilianweng.github.io
Exploration Strategies in Deep Reinforcement Learning
[Updated on 2020-06-17: Add “exploration via disagreement” in the “Forward Dynamics” section . Exploitation versus exploration is a critical topic in Reinforcement Learning. We’d like the RL agent to find the best solution as fast as possible. However, in the meantime, committing to solutions too quickly without enough exploration sounds pretty bad, as it could lead to local minima or total failure. Modern RL algorithms that optimize for the best returns can achieve
articlelilianweng.github.io
Neural Architecture Search
Although most popular and successful model architectures are designed by human experts, it doesn’t mean we have explored the entire network architecture space and settled down with the best option. We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.
articlelilianweng.github.io
Controllable Neural Text Generation
[Updated on 2021-02-01: Updated to version 2.0 with several work added and many typos fixed.] [Updated on 2021-05-26: Add P-tuning and Prompt Tuning in the “prompt design” section.] [Updated on 2021-09-19: Add “unlikelihood training” .]
articlelilianweng.github.io
Reducing Toxicity in Language Models
Large pretrained language models are trained over a sizable collection of online data. They unavoidably acquire certain toxic behavior and biases from the Internet. Pretrained language models are very powerful and have shown great success in many NLP tasks. However, to safely deploy them for practical real-world applications demands a strong safety control over the model generation process.
articlelilianweng.github.io
Contrastive Representation Learning
The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning .
articlelilianweng.github.io
What are Diffusion Models?
[Updated on 2021-09-19: Highly recommend this blog post on score-based generative modeling by Yang Song (author of several key papers in the references)]. [Updated on 2022-08-27: Added classifier-free guidance , GLIDE , unCLIP and Imagen . [Updated on 2022-08-31: Added latent diffusion model . [Updated on 2024-04-13: Added progressive distillation , consistency models , and the Model Architecture section .
articlelilianweng.github.io
Learning with not Enough Data Part 1: Semi-Supervised Learning
When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.
articlelilianweng.github.io
Learning with not Enough Data Part 2: Active Learning
This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.
articlelilianweng.github.io
Learning with not Enough Data Part 3: Data Generation
Here comes the Part 3 on learning with not enough data (Previous: Part 1 and Part 2 ). Let’s consider two approaches for generating synthetic data for training. Augmented data . Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a previous post on contrastive learning. For the sake of post completeness, I d
articlelilianweng.github.io
Generalized Visual Language Models
Processing images to generate text, such as image captioning and visual question-answering, has been studied for years. Traditionally such systems rely on an object detection network as a vision encoder to capture visual features and then produce text via a text decoder. Given a large amount of existing literature, in this post, I would like to only focus on one approach for solving vision language tasks, which is to extend pre-trained generalized language models to be capable of consuming visua
articlelilianweng.github.io
Large Transformer Model Inference Optimization
[Updated on 2023-01-24: add a small section on Distillation .] Large transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale. Why is it hard to run inference for large transformer models? Besides the increasing size of SoTA models, there are two main factor
articlelilianweng.github.io
The Transformer Family Version 2.0
Many new Transformer architecture improvements have been proposed since my last post on “The Transformer Family” about three years ago. Here I did a big refactoring and enrichment of that 2020 post — restructure the hierarchy of sections and improve many sections with more recent papers. Version 2.0 is a superset of the old version, about twice the length. Notations Symbol Meaning $d$ The model size / hidden state dimension / positional encoding size. $h$ The number of heads in
articlelilianweng.github.io
Prompt Engineering
Prompt Engineering , also known as In-Context Prompting , refers to methods for how to communicate with LLM to steer its behavior for desired outcomes without updating the model weights. It is an empirical science and the effect of prompt engineering methods can vary a lot among models, thus requiring heavy experimentation and heuristics. This post only focuses on prompt engineering for autoregressive language models, so nothing with Cloze tests, image generation or multimodality models. At its
articlelilianweng.github.io
LLM Powered Autonomous Agents
Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT , GPT-Engineer and BabyAGI , serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver. Agent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components: P
articlelilianweng.github.io
Adversarial Attacks on LLMs
The use of large language models in the real world has strongly accelerated by the launch of ChatGPT. We (including my team at OpenAI, shoutout to them) have invested a lot of effort to build default safe behavior into the model during the alignment process (e.g. via RLHF ). However, adversarial attacks or jailbreak prompts could potentially trigger the model to output something undesired. A large body of ground work on adversarial attacks is on images, and differently it operates in the continu
articlelilianweng.github.io
Thinking about High-Quality Human Data
[Special thank you to Ian Kivlichan for many useful pointers (E.g. the 100+ year old Nature paper “Vox populi”) and nice feedback. 🙏 ] High-quality data is the fuel for modern data deep learning model training. Most of the task-specific labeled data comes from human annotation, such as classification task or RLHF labeling (which can be constructed as classification format) for LLM alignment training. Lots of ML techniques in the post can help with data quality, but fundamentally hum
articlelilianweng.github.io
Diffusion Models for Video Generation
Diffusion models have demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task—using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because: It has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model. In comparison to text or images, it is
articlelilianweng.github.io
Extrinsic Hallucinations in LLMs
Hallucination in large language models usually refers to the model generating unfaithful, fabricated, inconsistent, or nonsensical content. As a term, hallucination has been somewhat generalized to cases when the model makes mistakes. Here, I would like to narrow down the problem of hallucination to cases where the model output is fabricated and not grounded by either the provided context or world knowledge. There are two types of hallucination: In-context hallucination: The model output should
articlelilianweng.github.io
Reward Hacking in Reinforcement Learning
Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function. With the rise of language models generalizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL
articlelilianweng.github.io
Why We Think
Special thanks to John Schulman for a lot of super valuable feedback and direct edits on this post. Test time compute ( Graves et al. 2016 , Ling, et al. 2017 , Cobbe et al. 2021 ) and Chain-of-thought (CoT) ( Wei et al. 2022 , Nye et al. 2021 ), have led to significant improvements in model performance, while raising many research questions. This post aims to review recent developments in how to effectively use test-time compute (i.e. “thinking time”) and why it helps.
articlelilianweng.github.io
Scaling Laws, Carefully
Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between $N$
articlelilianweng.github.io
How to Train Really Large Models on Many GPUs?
[Updated on 2022-03-13: add expert choice routing .] [Updated on 2022-06-10]: Greg and I wrote a shorted and upgraded version of this post, published on OpenAI Blog: “Techniques for Training Large Neural Networks”
articlelilianweng.github.io
Building OpenCode with Dax Raad
Stream the latest episode Listen and watch now on YouTube , Spotify , and Apple . See the episode transcript at the top of this page, and timestamps for the episode at the bottom. Brought to You by • Antithesis – if you’re using agentic workflows, you need to be extremely clear about what you’re building and how your system should behave. Antithesis brings specification and verification together, making your agents faster, smarter, and safer. And when you’re using A
articleGergely Orosz
Why is Meta destroying its engineering organization?
Hi – this is Gergely with a free issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of senior engineers and engineering leaders. Subscribe to get deepdives like this in your inbox, weekly: Subscribe now Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager . For two decades, Meta had a unique, high-performance eng
articleGergely Orosz
Context engineering with Dex Horthy
Stream the latest episode Listen and watch now on YouTube , Spotify , and Apple . See the episode transcript at the top of this page, and timestamps for the episode at the bottom. Brought to You by • Antithesis — with Antithesis, you can use AI agents to work on critical systems without worrying about correctness. Going far beyond code review, you can run your complete system in a hostile environment, analyze its behavior, and reproduce every issue perfectly. Teams like Jane Street,
articleGergely Orosz
The Pulse: Grok’s CLI caught uploading all your local files to the cloud
The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me. Today, we cover: Grok’s CLI uploaded all your local files to the cloud, then got caught . Just as devs were starting to use the Grok CLI – thanks to the capable Grok 4.5 coding model – they discover… Read more
articleGergely Orosz
Slow down to speed up: so much has changed in 6 months’ time
Scheduling note: there will be no edition of The Pulse on Thursday as I’m in San Francisco for the next week and a half, visiting AI labs and startups, and attending the AI Engineer World Fair from next Monday. For the podcast and Tuesday articles, it’s business as usual. Three weeks ago, at Craft Conference , in Budapest, Hungary, I opened the event with a keynote titled ‘Slow Down to Speed Up’. As with most of my talks, it came together in stages, including with some in
articleGergely Orosz
Impressions from visiting OpenAI, Anthropic, & Cursor
Scheduling note: this week, I’m in San Francisco at the AI Engineer’s World Fair , so there won’t be an edition of The Pulse on Thursday. However, tomorrow (Wednesday) there will be a special podcast episode – the lengthiest, most detailed one yet – with software engineering legend, Kent Beck. In recent days, I’ve visited the offices of OpenAI, Anthropic, and Cursor, in San Francisco. Onsite, I talked with software folks working on the model side to learn more
articleGergely Orosz
Satya Nadella — Microsoft’s AGI plan & quantum breakthrough
Satya Nadella on: Why he doesn’t believe in AGI but does believe in 10% economic growth; Microsoft’s new topological qubit breakthrough and gaming world models; Whether Office commoditizes LLMs or the other way around. Watch on Youtube ; listen on Apple Podcasts or Spotify . ---------- Sponsors Scale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale’s Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilitie
podcastDwarkesh Patel
AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo
Scott and Daniel break down every month from now until the 2027 intelligence explosion. Scott Alexander is author of the highly influential blogs Slate Star Codex and Astral Codex Ten . Daniel Kokotajlo resigned from OpenAI in 2024, rejecting a non-disparagement clause and risking millions in equity to speak out about AI safety. We discuss misaligned hive minds, Xi and Trump waking up, and automated Ilyas researching AI progress. I came in skeptical, but I learned a tremendous amount by bouncing
podcastDwarkesh Patel
AGI is still 30 years away — Ege Erdil & Tamay Besiroglu
Ege Erdil and Tamay Besiroglu have 2045+ timelines, think the whole "alignment" framing is wrong, don't think an intelligence explosion is plausible, but are convinced we'll see explosive economic growth (economy literally doubling every year or two). This discussion offers a totally different scenario than my recent interview with Scott and Daniel . Ege and Tamay are the co-founders of Mechanize (disclosure - I’m an angel investor), a startup dedicated to fully automating work. Before founding
podcastDwarkesh Patel
Mark Zuckerberg — AI will write most Meta code in 18 months
Zuck on: * Llama 4, benchmark gaming * Intelligence explosion, business models for AGI * DeepSeek/China, export controls, & Trump * Orion glasses, AI relationships, and preventing reward-hacking from our tech. Watch on Youtube ; listen on Apple Podcasts and Spotify . ---------- SPONSORS * Scale is building the infrastructure for safer, smarter AI. Scale’s Data Foundry gives major AI labs access to high-quality data to fuel post-training, while their public leaderboards help assess model capabili
podcastDwarkesh Patel
Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
New episode with my good friends Sholto Douglas & Trenton Bricken . Sholto focuses on scaling RL and Trenton researches mechanistic interpretability, both at Anthropic. We talk through what’s changed in the last year of AI research; the new RL regime and how far it can scale; how to trace a model’s thoughts; and how countries, workers, and students should prepare for AGI. See you next year for v3. Here’s last year’s episode , btw. Enjoy! Watch on YouTube ; listen on Apple Podcasts or Spotify . -
podcastDwarkesh Patel
Why I don’t think AGI is right around the corner
I’ve had a lot of discussions on my podcast where we haggle out timelines to AGI. Some guests think it’s 20 years away - others 2 years . Here’s an audio version of where my thoughts stand as of June 2025. If you want to read the original post, you can check it out here . Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
podcastDwarkesh Patel
Fully autonomous robots are much closer than you think – Sergey Levine
Sergey Levine , one of the world’s top robotics researchers and co-founder of Physical Intelligence , thinks we’re on the cusp of a “self-improvement flywheel” for general-purpose robots. His median estimate for when robots will be able to run households entirely autonomously? 2030. If Sergey’s right, the world 5 years from now will be an insanely different place than it is today. This conversation focuses on understanding how we get there: we dive into foundation models for robotics, and how we
podcastDwarkesh Patel
Richard Sutton – Father of RL thinks LLMs are a dead end
Richard Sutton is the father of reinforcement learning, winner of the 2024 Turing Award, and author of The Bitter Lesson. And he thinks LLMs are a dead end. After interviewing him, my steel man of Richard’s position is this: LLMs aren’t capable of learning on-the-job, so no matter how much we scale, we’ll need some new architecture to enable continual learning. And once we have it, we won’t need a special training phase — the agent will just learn on-the-fly, like all humans, and indeed, like al
podcastDwarkesh Patel
Some thoughts on the Sutton interview
I have a much better understanding of Sutton’s perspective now. I wanted to reflect on it a bit. (00:00:00) - The steelman (00:02:42) - TLDR of my current thoughts (00:03:22) - Imitation learning is continuous with and complementary to RL (00:08:26) - Continual learning (00:10:31) - Concluding thoughts Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
podcastDwarkesh Patel
Andrej Karpathy — AGI is still a decade away
The Andrej Karpathy episode. During this interview, Andrej explains why reinforcement learning is terrible (but everything else is much worse), why AGI will just blend into the previous ~2.5 centuries of 2% GDP growth, why self driving took so long to crack, and what he sees as the future of education. It was a pleasure chatting with him. Watch on YouTube ; read the transcript . Sponsors * Labelbox helps you get data that is more detailed, more accurate, and higher signal than you could get by d
podcastDwarkesh Patel
Ilya Sutskever — We're moving from the age of scaling to the age of research
Ilya & I discuss SSI’s strategy, the problems with pre-training, how to improve the generalization of AI models, and how to ensure AGI goes well. Watch on YouTube ; read the transcript . Sponsors * Gemini 3 is the first model I’ve used that can find connections I haven’t anticipated. I recently wrote a blog post on RL’s information efficiency, and Gemini 3 helped me think it all through. It also generated the relevant charts and ran toy ML experiments for me with zero bugs. Try Gemini 3 today at
podcastDwarkesh Patel
Dario Amodei — "We are near the end of the exponential"
Dario Amodei thinks we are just a few years away from AGI — or as he puts it, from having “a country of geniuses in a data center”. In this episode, we discuss what to make of the scaling hypothesis in the current RL regime, why task-specific RL might lead to generalization, and how AI will diffuse throughout the economy. We also dive into Anthropic’s revenue projections, compute commitments, path to profitability, and more. Watch on YouTube ; read the transcript . Sponsors * Labelbox can get yo
podcastDwarkesh Patel
Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute
Dylan Patel , founder of SemiAnalysis , provides a deep dive into the 3 big bottlenecks to scaling AI compute: logic, memory, and power. And walks through the economics of labs, hyperscalers, foundries, and fab equipment manufacturers. Learned a ton about every single level of the stack. Enjoy! Watch on YouTube ; read the transcript . Sponsors * Mercury has already saved me a bunch of time this tax season. Last year, I used Mercury to request W-9s from all the contractors I worked with. Then, wh
podcastDwarkesh Patel
Terence Tao – Kepler, Newton, and the true nature of mathematical discovery
We begin the episode with the absolutely ingenious and surprising way in which Kepler discovered the laws of planetary motion. People sometimes say that AI will make especially fast progress at scientific discovery because of tight verification loops. But the story of how we discovered the shape of our solar system shows how the verification loop for correct ideas can be decades (or even millennia) long. During this time, what we know today as the better theory can actually make worse prediction
podcastDwarkesh Patel
Reiner Pope – The math behind how LLMs are trained and served
Did a very different format with Reiner Pope - a blackboard lecture where he walks through how frontier LLMs are trained and served. It’s shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there – it’s really worth it. There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real de
podcastDwarkesh Patel
The next big breakthrough will be AIs learning on the job
Read it here . Thanks to Mercury for sponsoring this essay. Mercury has automated basically my entire bill pay process for my business. I just give contractors a dedicated email address, and when they send an invoice, Mercury automatically creates a draft payment for me to review. I no longer have to hunt through my inbox for invoices or deal with messy spreadsheets to track my bills. Mercury handles it all. Learn more at mercury.com Timestamps: (00:00:00) – The big research bet the labs are mak
podcastDwarkesh Patel
Eric Jang – Building AlphaGo from scratch
Eric Jang walks through how to build AlphaGo from scratch, but with modern AI tools. Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn. Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it co
podcastDwarkesh Patel
#452 – Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity
Dario Amodei is the CEO of Anthropic, the company that created Claude. Amanda Askell is an AI researcher working on Claude’s character and personality. Chris Olah is an AI researcher working on mechanistic interpretability. Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep452-sc See below for timestamps, transcript, and to give feedback, submit questions, contact Lex, etc. Transcript: https://lexfridman.com/dario-amodei-transcript CONTACT LEX: Feedback &#
podcastlexfridman.com
#459 – DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters
Dylan Patel is the founder of SemiAnalysis, a research & analysis company specializing in semiconductors, GPUs, CPUs, and AI hardware. Nathan Lambert is a research scientist at the Allen Institute for AI (Ai2) and the author of a blog on AI called Interconnects. Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep459-sc See below for timestamps, transcript, and to give feedback, submit questions, contact Lex, etc. Transcript: https://lexfridman.com/deepseek-dy
podcastlexfridman.com
#475 – Demis Hassabis: Future of AI, Simulating Reality, Physics and Video Games
Demis Hassabis is the CEO of Google DeepMind and Nobel Prize winner for his groundbreaking work in protein structure prediction using AI. Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep475-sc See below for timestamps, transcript, and to give feedback, submit questions, contact Lex, etc. Transcript: https://lexfridman.com/demis-hassabis-2-transcript CONTACT LEX: Feedback – give feedback to Lex: https://lexfridman.com/survey AMA – submit questions,
podcastlexfridman.com
#490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI
Nathan Lambert and Sebastian Raschka are machine learning researchers, engineers, and educators. Nathan is the post-training lead at the Allen Institute for AI (Ai2) and the author of The RLHF Book. Sebastian Raschka is the author of Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch). Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep490-sc See below for timestamps, transcript, and to give feedback, submit questions, contact L
podcastlexfridman.com
Adobe Agentic Sites (Audience of One)
An experimental Adobe concept where websites assemble themselves in real time around each visitor's intent. Instead of static pages, an LLM retrieves from a company's existing content and composes a personalized page on the fly for individual users and even AI agents.
articleRichard MacManus
AINews
AINews is a newsletter and searchable archive that recaps daily AI developments across Twitter, Reddit, and Discord, summarizing model releases, tooling launches, and research. It's now published as a section of Latent Space with configurable email frequencies.
articlewww.latent.space
AI Engineer World's Fair 2026 Recap
A recap article covering five major trends from the AI Engineer World's Fair 2026, exploring how AI engineering has shifted from building individual agents toward engineering the reliable systems, loops, and skills that surround them. It's a useful state-of-the-field overview for developers tracking where coding agents and agentic infrastructure are heading.
articleRichard MacManus
Against "Brain Damage"
An essay by Ethan Mollick (Wharton Generative AI Lab) exploring how AI can help or hurt human thinking across learning, creativity, and teamwork. It debunks the 'AI causes brain damage' fear and offers practical strategies plus prompts for using AI without outsourcing your thinking.
articleEthan Mollick
On Working with Wizards
An essay by Ethan Mollick exploring how the nature of working with AI is shifting from collaborative 'co-intelligence' to a 'wizard' model, where AI produces sophisticated outputs through opaque processes that users can only verify, not shape.
articleEthan Mollick
Real AI Agents and Real Work
An analytical essay by Ethan Mollick (One Useful Thing) examining how AI agents have crossed a threshold into performing economically valuable tasks. It explores OpenAI's expert task benchmark, uses Claude Sonnet 4.5 to reproduce academic research, and argues that the future of work depends on how thoughtfully we deploy AI.
articleEthan Mollick
The Bitter Lesson versus The Garbage Can
An essay exploring whether AI agents can navigate messy, undocumented organizational processes by training on outcomes rather than encoding human expertise. It connects Richard Sutton's 'Bitter Lesson' and the 'Garbage Can Model' of organizations to argue enterprise AI adoption may not require mapping chaotic workflows.
articleEthan Mollick
GPT-5
GPT-5 is OpenAI's flagship AI model that automatically selects the right reasoning level for each task and proactively takes action on vague prompts. It can generate startup plans, build functional apps, create documents, and 'just do stuff' with minimal user guidance.
articleEthan Mollick
Mass Intelligence
An analytical essay by Ethan Mollick (One Useful Thing) exploring how powerful AI is becoming as accessible as a Google search, covering the economics, ease of use, and societal implications of a billion people gaining access to advanced models like GPT-5 and Gemini's 'nano banana' image generator.
articleEthan Mollick
A Guide to Which AI to Use in the Agentic Era
An explanatory guide that breaks down how to choose AI tools in 2025 by understanding three layers: models (the AI brains), apps (products you use), and harnesses (systems that let AI take actions and complete tasks). It walks through the current landscape of ChatGPT, Claude, and Gemini and their various coding agents and desktop tools.
articleEthan Mollick
The Shape of AI: Jaggedness, Bottlenecks and Salients
An essay by Ethan Mollick analyzing why AI capabilities improve unevenly (the 'jagged frontier') and how single bottlenecks hold back entire systems until they suddenly break through. Uses Google's Nano Banana Pro image model as a case study for how solving one weakness can unlock a wave of new capabilities.
articleEthan Mollick
Giving your AI a Job Interview
An essay by Ethan Mollick arguing that organizations should evaluate AI models like job candidates rather than relying on public benchmarks. It explains why standardized benchmarks fall short and offers practical approaches like vibes-testing, real-world task benchmarking (GDPval), and probing an AI's judgment on ambiguous business questions.
articleEthan Mollick
The Shape of the Thing
An essay by Ethan Mollick analyzing the current state of AI, arguing that we've entered an era of managing AI agents rather than working alongside them. It explores exponential AI capability gains, radical new organizational structures like AI-only 'software factories,' and the looming reality of recursive self-improvement.
articleEthan Mollick
Claude Cowork with Dispatch
An article examining how AI interfaces shape usability, highlighting Anthropic's Claude Cowork (a Claude Code variant for knowledge workers) and its Dispatch feature, which lets you control an AI agent on your desktop from your phone via QR code. It explores why chatbot interfaces impose cognitive load and how specialized or on-demand interfaces unlock AI's real capabilities.
articleEthan Mollick
Choosing to Stay Human
An essay by Ethan Mollick exploring how to use AI thoughtfully for writing and learning without surrendering the cognitive work that builds real skill. It draws on research studies to distinguish AI use that shortcuts thinking from AI use that strengthens it.
articleEthan Mollick
Co-Existence Book Website
The promotional website for Ethan Mollick's book 'Co-Existence,' notable for including a dedicated version of the page addressed to AI agents that read on behalf of human users. The site was built with Claude Code and A/B tested against dozens of AI models to optimize how it presents to both humans and AI readers.
articleEthan Mollick
The 100-person AI lab that became Anthropic and Google's secret weapon | Edwin Chen (Surge AI)
Edwin Chen is the founder and CEO of Surge AI, the company that teaches AI what’s good vs. what’s bad, powering frontier labs with elite data, environments, and evaluations. Surge surpassed $1 billion in revenue with under 100 employees last year, completely bootstrapped—the fastest company in history to reach this milestone. Before founding Surge, Edwin was a research scientist at Google, Facebook, and Twitter and studied mathematics, computer science, and linguistics at MIT. We discuss: 1. How
podcastLenny Rachitsky
Why humans are AI’s biggest bottleneck (and what’s coming in 2026) | Alexander Embiricos (OpenAI Codex Product Lead)
Alexander Embiricos leads product on Codex, OpenAI’s powerful coding agent, which has grown 20x since August and now serves trillions of tokens weekly. Before joining OpenAI, Alexander spent five years building a pair programming product for engineers. He now works at the frontier of AI-led software development, building what he describes as a software engineering teammate—an AI agent designed to participate across the entire development lifecycle. We discuss: 1. Why Codex has grown 20x since la
podcastLenny Rachitsky
The coming AI security crisis (and what to do about it) | Sander Schulhoff
Sander Schulhoff is an AI researcher specializing in AI security, prompt injection, and red teaming. He wrote the first comprehensive guide on prompt engineering and ran the first-ever prompt injection competition, working with top AI labs and companies. His dataset is now used by Fortune 500 companies to benchmark their AI systems security, he’s spent more time than anyone alive studying how attackers break AI systems, and what he’s found isn’t reassuring: the guardrails companies are buying do
podcastLenny Rachitsky
Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon
Aishwarya Naresh Reganti and Kiriti Badam have helped build and launch more than 50 enterprise AI products across companies like OpenAI, Google, Amazon, and Databricks. Based on these experiences, they’ve developed a small set of best practices for building and scaling successful AI products. The goal of this conversation is to save you and your team a lot of pain and suffering. We discuss: 1. Two key ways AI products differ from traditional software, and why that fundamentally changes how they
podcastLenny Rachitsky
The non-technical PM’s guide to building with Cursor | Zevi Arnovitz (Meta)
Zevi Arnovitz is a product manager at Meta with no technical background who has figured out how to build and ship real products using AI. His engineering team at Meta asks him to teach them how he does what he does. In this episode, Zevi breaks down his complete AI workflow that allows non-technical people to build sophisticated products with Cursor. We discuss: 1. The complete AI workflow that lets non-technical people build real products in Cursor 2. How to use multiple AI models for different
podcastLenny Rachitsky
The rise of the professional vibe coder (a new AI-era job) | Lazar Jovanovic (Professional Vibe Coder)
Lazar Jovanovic is a full-time professional vibe coder at Lovable. His job is to build both internal tools and customer-facing products purely using AI, while not having a coding background. In this conversation, he breaks down the tactics, workflows, and framework that let him ship production-quality products using only AI. We discuss: 1. Why having no coding background can be an advantage when building with AI 2. Why most of your time should go to planning and chat mode, not prompting 3. What
podcastLenny Rachitsky
“Engineers are becoming sorcerers” | The future of software development with OpenAI’s Sherwin Wu
Sherwin Wu leads engineering for OpenAI’s API platform, where roughly 95% of engineers use Codex, often working with fleets of 10 to 20 parallel AI agents. We discuss: 1. What OpenAI did to cut code review times from 10-15 minutes to 2-3 minutes 2. How AI is changing the role of managers 3. Why the productivity gap between AI power users and everyone else is widening 4. Why “models will eat your scaffolding for breakfast” 5. Why the next 12 to 24 months are a rare window where engineers can leap
podcastLenny Rachitsky
Head of Claude Code: What happens after coding is solved | Boris Cherny
Boris Cherny is the creator and head of Claude Code at Anthropic. What began as a simple terminal-based prototype just a year ago has transformed the role of software engineering and is increasingly transforming all professional work. We discuss: 1. How Claude Code grew from a quick hack to 4% of public GitHub commits, with daily active users doubling last month 2. The counterintuitive product principles that drove Claude Code’s success 3. Why Boris believes coding is “solved” 4. The latent dema
podcastLenny Rachitsky
The design process is dead. Here’s what’s replacing it. | Jenny Wen (head of design at Claude)
Jenny Wen leads design for Claude at Anthropic. Prior to this, she was Director of Design at Figma, where she led the teams behind FigJam and Slides. Before that, she was a designer at Dropbox, Square, and Shopify. — We discuss: 1. Why the classic discovery → mock → iterate design process is becoming obsolete 2. What a day in the life of a designer at Anthropic looks like, including her AI tool stack 3. Whether AI will eventually surpass humans in taste and judgment 4. Why Jenny left a director
podcastLenny Rachitsky
From skeptic to true believer: How OpenClaw changed my life | Claire Vo
Claire Vo is the host of our sister podcast, “How I AI,” a former product executive and engineer, and founder of an AI startup called ChatPRD. Claire now runs her business, podcast, and family life with the help of nine OpenClaw agents running on multiple Mac Minis and old laptops. In this episode, Claire shares her journey from OpenClaw skeptic (it deleted her family calendar the first time she tried it) to true believer, and gives a masterclass in using AI agents in real life. We discuss: 1. T
podcastLenny Rachitsky
An AI state of the union: We’ve passed the inflection point, dark factories are coming, and automation timelines | Simon Willison
Simon Willison is a prolific independent software developer, a blogger, and one of the most visible and trusted voices on the impact AI is having on builders. He co-created Django, the web framework that powers Instagram, Pinterest, and tens of thousands of other websites. He coined the term “prompt injection,” popularized the terms “AI slop” and “agentic engineering,” and has built over 100 open source projects, including Datasette, a data analysis tool used by investigative journalists worldwi
podcastLenny Rachitsky
Head of Growth (Anthropic): “Claude is growing itself at this point” | Amol Avasare
Amol Avasare is Head of Growth at Anthropic, which is going through the most unprecedented growth trajectory in history—scaling from $1 billion to over $19 billion in ARR in just 14 months. Previously, Amol worked on the growth teams at Mercury and MasterClass. Before that he was a founder, and he cold emailed his way into the Anthropic role when no job listing existed. Most remarkably, he overcame a traumatic brain injury from a Muay Thai match that meant he couldn't work for nearly a year. In
podcastLenny Rachitsky
How Anthropic’s product team moves faster than anyone else | Cat Wu (Head of Product, Claude Code)
Cat Wu is Head of Product for Claude Code and Cowork at Anthropic, building one of the most important AI products of this generation. Before joining Anthropic, Cat spent years as an engineer and briefly worked in VC. Today, she’s interviewing hundreds of product managers who are trying to break into AI—and seeing firsthand what separates those who thrive from those who fall behind. We discuss: 1. How Anthropic’s shipping cadence went from months to weeks to days 2. The emerging skills PMs need t
podcastLenny Rachitsky
Why cultivating agency matters more than cultivating skills in the AI era | Max Schoening (Head of Product, Notion)
Max Schoening is head of product at Notion, where he’s been especially effective at getting designers and PMs to ship code, prototype in the terminal, and launch extremely successful AI products. He was previously a PM at Google, ran design at Heroku, was VP of Design (and a part-time engineer) at GitHub, and is a two-time founder. He’s one of the most AI-forward product leaders out there and one of the deepest thinkers on how AI changes how we build and use software. We discuss: 1. What’s most
podcastLenny Rachitsky
The AI paradox: More automation, more humans, more work | Dan Shipper
Dan Shipper is the co-founder and CEO of Every, a media and software company that’s become a living laboratory for the future of work. Everyone at his company of about 30 people is an AI early adopter; from editors to ops people, they use AI to do much of their work, giving Every a unique lens into where the world is heading. A year ago on this show, Dan predicted that people were sleeping on Claude Code for nontechnical work, which proved to be remarkably prescient. Today he’s back with another
podcastLenny Rachitsky
What happens after coding is solved? | Fiona Fung (Manager of the Claude Code and Cowork Teams)
Fiona Fung leads the teams behind Claude Code and Cowork at Anthropic (overseeing Boris Cherny and the entire engineering and PM team). Before Anthropic, she spent 11 years at Microsoft building Visual Studio and TypeScript and then moved to Meta, where she started Facebook Marketplace (now generating over $100 billion in GMV annually), worked on Meta’s first smart glasses and AR glasses, and led infrastructure, growth, integrity, and safety teams at Instagram. She’s been an engineer for over 25
podcastLenny Rachitsky
Beyond Standard LLMs
An in-depth educational article by Sebastian Raschka exploring alternative LLM architectures beyond standard autoregressive transformers, including linear attention hybrids, text diffusion models, code world models, and small recursive transformers. It breaks down architectures like Qwen3-Next, Kimi Linear, MiniMax-M1/M2, and DeepSeek V3.2 with technical explanations and code illustrations.
articleSebastian Raschka, PhD
From GPT-2 to gpt-oss: Analyzing the Architectural Advances
An in-depth technical article breaking down the architectural design choices in OpenAI's gpt-oss-120b and gpt-oss-20b open-weight models, comparing them against GPT-2 and Qwen3. It walks through MoE, RoPE, GQA, sliding-window attention, RMSNorm, MXFP4 optimization, and more.
articleSebastian Raschka, PhD
Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)
An educational article that explains the four primary methods for evaluating large language models — multiple-choice benchmarks, verifiers, leaderboards, and LLM judges — with from-scratch PyTorch code examples. It helps readers build a mental map for interpreting benchmarks, leaderboards, and research papers.
articleSebastian Raschka, PhD
Understanding and Implementing Qwen3 From Scratch
An in-depth educational article that walks through the Qwen3 LLM architecture and reimplements it from scratch in pure PyTorch. It covers both the dense and Mixture-of-Experts variants, helping readers understand how modern open-weight models work under the hood.
articleSebastian Raschka, PhD
A Technical Tour of the DeepSeek Models from V3 to V3.2
An in-depth technical article walking through the architectural evolution of DeepSeek's open-weight LLMs from V3 to V3.2, explaining concepts like Multi-Head Latent Attention, RLVR reasoning training, and the new DeepSeek Sparse Attention mechanism.
articleSebastian Raschka, PhD
LLM Research Papers: The 2025 List
A topic-organized collection of 200+ large language model research papers from the first half of 2025, curated by Sebastian Raschka. Papers are grouped by themes like reasoning models, reinforcement learning, and multimodal models for easy reference.
articleSebastian Raschka, PhD
The Big LLM Architecture Comparison
An in-depth technical article that compares the architectural designs of modern flagship open-weight LLMs, from DeepSeek-V3 and OLMo 2 to Kimi K2 and Gemma. It breaks down design choices like Multi-Head Latent Attention, Mixture-of-Experts, normalization placement, and QK-Norm.
articleSebastian Raschka, PhD
Coding LLMs from the Ground Up
A ~15-hour video course by Sebastian Raschka that teaches you how to build a large language model from scratch in Python — covering tokenization, attention mechanisms, pretraining, and finetuning. It serves as standalone companion content to his 'Build a Large Language Model (From Scratch)' book.
articleSebastian Raschka, PhD
Understanding and Coding the KV Cache in LLMs from Scratch
A hands-on technical tutorial that explains how key-value (KV) caches speed up LLM inference, walking through a from-scratch, human-readable PyTorch implementation. It compares text generation with and without caching and demonstrates a ~5x speedup on a 124M parameter model.
articleSebastian Raschka, PhD
LLM Research Papers: The 2026 List
A curated, categorized reading list of notable LLM research papers published from January to May 2026, organized into topics like architecture, reasoning, reinforcement learning, and agent systems. It's a reference resource for researchers and practitioners who want to track the field's developments.
articleSebastian Raschka, PhD
Understanding LLM Architectures Workflow
A learning-oriented walkthrough of how to reverse-engineer and understand new open-weight LLM architectures by inspecting Hugging Face config files and reference implementations rather than relying on sparse technical papers. Useful for ML practitioners who want to deeply understand how modern model architectures actually work.
articleSebastian Raschka, PhD
Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention
An in-depth technical article analyzing recent open-weight LLM architecture innovations focused on long-context efficiency, covering KV sharing and per-layer embeddings in Gemma 4, attention budgeting in Laguna XS.2, compressed convolutional attention in ZAYA1-8B, and DeepSeek V4's mHC design. Useful for ML engineers and researchers who want to understand how modern transformers reduce KV cache size and attention costs.
articleSebastian Raschka, PhD
Categories of Inference-Time Scaling for Improved LLM Reasoning
An in-depth educational article that categorizes and explains inference-time scaling techniques for improving LLM reasoning, covering methods like chain-of-thought prompting, self-consistency, best-of-N ranking, rejection sampling, self-refinement, and search over solution paths. It's a companion piece to a book chapter on building reasoning models from scratch.
articleSebastian Raschka, PhD
Controlling Reasoning Effort in LLMs
An in-depth technical article explaining how large language models learn to support low-, medium-, and high-effort reasoning modes. It covers reasoning model training via RLVR, think tokens, on/off reasoning switches, and how reasoning effort settings work in models like GPT-5, Qwen3, and gpt-oss.
articleSebastian Raschka, PhD
A Visual Guide to Attention Variants in Modern LLMs
An educational article and visual gallery explaining the evolution of attention mechanisms in modern LLMs, from Multi-Head Attention (MHA) and Grouped-Query Attention (GQA) to Multi-Head Latent Attention (MLA) and sparse/hybrid architectures. Each concept is illustrated with visual model cards and real-world example architectures.
articleSebastian Raschka, PhD
Mini Coding Agent
A minimal but fully working coding agent implemented in pure Python, designed to demonstrate the six core building blocks of coding agents like Claude Code and Codex CLI. It serves as an educational reference for understanding how agent harnesses use tools, memory, and repo context to make LLMs more capable at software tasks.
articleSebastian Raschka, PhD
The State Of LLMs 2025
An in-depth year-in-review article covering the major developments in large language models throughout 2025, including DeepSeek R1, RLVR, GRPO, inference-time scaling, and architectural trends, plus predictions for 2026 and 2027. Written for practitioners and researchers who want a technically grounded synthesis of the year's LLM progress.
articleSebastian Raschka, PhD
Predictive Human Preference: From Model Ranking to Model Routing
A challenge of building AI applications is choosing which model to use. What if we don’t have to? What if we can predict the best model for any prompt? Predictive human preference aims to predict which model users might prefer for a specific query. Human preference has emerged to be both the Northstar and a powerful tool for AI model development. Human preference guides post-training techniques including RLHF and DPO . Human preference is also used to rank AI models, as used by LMSYS’s Chatbot A
articlehuyenchip.com
What I learned from looking at 900 most popular open source AI tools
[ Hacker News discussion , LinkedIn discussion , Twitter thread ] Update (Feb 2026) : The full list of open source AI repos is hosted at Good AI List , updated daily. It’s balooned to 15K repos, and you can submit missing repos. You can also find some of them on my cool-llm-repos list on GitHub. Four years ago, I did an analysis of the open source ML ecosystem . Since then, the landscape has changed, so I revisited the topic. This time, I focused exclusively on the stack around foundation models
articlehuyenchip.com
Building A Generative AI Platform
After studying how companies deploy generative AI applications, I noticed many similarities in their platforms. This post outlines the common components of a generative AI platform, what they do, and how they are implemented. I try my best to keep the architecture general, but certain applications might deviate. This is what the overall architecture looks like. This is a pretty complex system. This post will start from the simplest architecture and progressively add more components. In its simpl
articlehuyenchip.com
Agents
Intelligent agents are considered by many to be the ultimate goal of AI. The classic book by Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach (Prentice Hall, 1995), defines the field of AI research as “ the study and design of rational agents. ” The unprecedented capabilities of foundation models have opened the door to agentic applications that were previously unimaginable. These new capabilities make it finally possible to develop autonomous, intelligent agents to ac
articlehuyenchip.com
LLM Course
A comprehensive course for getting into Large Language Models (LLMs), featuring structured roadmaps and hands-on Google Colab notebooks. It walks learners through the full LLM stack from fundamentals to advanced topics. Ideal for developers and researchers looking to build practical skills in modern AI.
postx.com
post
Learning on the Shop Floor
A personal essay by Shopify CEO Tobi Lütke about his apprenticeship in Germany after dropping out of school at 16 to work at a Siemens subsidiary. The piece reflects on how hands-on, basement-level hacker culture shaped his technical and entrepreneurial thinking. It's a rare autobiographical look at the unconventional path behind one of tech's most prominent founders.
postx.com
post
The Reverse Information Paradox
An article shared by Microsoft CEO Satya Nadella exploring how firms should protect their core intellectual property in the age of AI and intelligence. It references Nobel Prize-winning economist Kenneth Arrow's information paradox and examines its inverse implications for modern businesses.
postx.com
Billion Dollar PDFs
Billion Dollar PDFs is a curated collection of exceptional writing and documents available online. It aggregates high-quality PDFs — likely including influential business, finance, and intellectual works — making them easy to discover and access in one place.
postx.com
post
AI Makes It Easier to Build the Wrong Product
An article by Grey (@greyluox) arguing that while AI has dramatically reduced the cost of producing software, it has not reduced the cost of building the wrong thing. The piece explores how teams are responding to cheaper implementation by making larger bets, potentially amplifying mistakes. It challenges builders to think critically about product direction before leveraging AI to accelerate execution.
postx.com
post
AI's Biggest Winners Have the Lowest Margins
An article arguing that the true winners of the AI era will be low-margin businesses — not high-tech companies — because AI dramatically reduces their operational costs and unlocks massive efficiency gains. It reframes the AI competitive landscape around cost structure rather than engineering headcount or data budgets.
postx.com
post
What Is a Software Factory?
An article by Chamath Palihapitiya exploring the concept of a 'software factory' — a modern approach to building software at scale. It draws from real-world experience at 8090, where the team reverse-engineered and replaced an 18-million-line COBOL/Assembly billing engine. It examines how AI-powered development workflows can modernize legacy systems at enterprise scale.
postx.com
post
Software Factory Builder
A personal software factory system built to automate software creation workflows with simplicity as a core principle. The creator shares how it works, what they learned, and how others can replicate it in an afternoon. It's a practical guide-plus-system for anyone looking to systematize AI-assisted software development.
postx.com
post
AI-pilling our company
An article by Neil Rahilly about adopting AI tools across a company, framed through the lens of the 1968 study on software engineer productivity variance. It explores why AI adoption matters and how it can dramatically amplify individual and team output.
postx.com
OpenAI Codex lead on the new shape of product work | Andrew Ambrosino
Andrew Ambrosino leads development of the Codex desktop app at OpenAI. Nearly 100% of OpenAI employees—not just engineers—now use Codex weekly. A lifelong builder with a background spanning engineering, design, product management, and founding companies, he is now responsible for turning the Codex desktop experience into what he calls “the best desktop app that has ever existed, full stop.” In our in-depth conversation, we discuss: 1. Why AI has completely flipped the product development process
podcastLenny Rachitsky
The Pulse: What can we learn from Bun’s rapid Rust rewrite with AI?
The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me. Today, we cover: Bun’s Rust rewrite with Fable: what can we learn? To a sceptic, spending $165K to migrate Bun from Zig to Rust sounds very expensive. But to a realist, shortening a 1-2 year migration down to 11 days opens amazing new opportunities for devs. However, a thoroughly-tested project is required to pull it off. Anthropic&
articleGergely Orosz
article
Harness Engineering for Self-Improvement
The concept of recursive self-improvement (RSI) dates back to I. J. Good (1965) , where he defined an “ultraintelligent machine” as a system that can surpass humans in all intellectual activities and design better machines to improve itself. Yudkowsky (2008) used the phrase “recursive self-improvement” for a specific feedback loop: an AI uses its current intelligence to improve the cognitive machinery that produces its intelligence. This feedback loop in modern AI may ind
articlelilianweng.github.io
Common pitfalls when building generative AI applications
As we’re still in the early days of building applications with foundation models, it’s normal to make mistakes. This is a quick note with examples of some of the most common pitfalls that I’ve seen, both from public case studies and from my personal experience. Because these pitfalls are common, if you’ve worked on any AI product, you’ve probably seen them before. 1. Use generative AI when you don't need generative AI Every time there’s a new technology, I can hear the collective sigh of senior
articlehuyenchip.com
Latest open artifacts (#22): Zyphra, Cohere, and Poolside are expanding the breadth of the ecosystem
A trend we continue to see in open model releases is that the ecosystem is becoming more diverse, with an increasing number of organizations releasing a wide range of models. A year ago, open artifacts and the open model landscape more broadly were dominated by a handful of (Chinese) players. This has shifted, with us increasingly featuring more niche companies all over the world. While it is hard to know the exact motivations of the companies themselves, we can broadly observe the following cat
articleFlorian Brand
The twilight of the chatbots
If you feel like things are accelerating in AI, you are probably right. Better AI models from the leading American AI labs have been releasing more quickly than ever (though government interventions stopped access temporarily to two of the most powerful models, Claude Fable and GPT-5.6). But it isn't just release timing. The evidence points to accelerating capability gains as well (though the frontier stays jagged, and AIs remain weak in many places). This is especially obvious when we look at t
articleEthan Mollick
Using Local Coding Agents
Many people reached out to me in the past asking about my local agent stack as well as how I set up my local agent stack. So, I thought it might be useful to put together a little tutorial on how to set up a local (coding) agent using open-source tools and open-weight LLMs. Figure 1: Overview of the local stack, that is, a coding agent harness that uses a local model hosted through an inference engine / runtime server. This article is a tutorial on setting up a production-ready coding agent with
articleSebastian Raschka, PhD
To learn anything, first unlearn school
An article by Zara Zhang exploring how to shed school-conditioned mental blocks around learning, using language learning as a case study. It challenges the 'bad at languages' self-label and offers a reframe for adult learners returning to skills they abandoned. Written for anyone who wants to learn something new but feels held back by past academic failure.
postx.com
How to Land a Job at a Frontier Lab
A guide by Vlad Feinberg (Gemini Pretraining Area TL at Google DeepMind) on how to get hired at a top AI research lab like Anthropic, OpenAI, xAI, or DeepMind. It offers insider perspective from someone actively working at the frontier of AI research and development.
postx.com
Getting Started with Loops
An article by ClaudeDevs explaining the concept of 'designing loops' for coding agents, a emerging paradigm shift from traditional prompting. It helps developers building with Claude AI understand how to structure agent workflows for more reliable and effective code generation.
postx.com
AI Policy Templates
A public-domain collection of generative AI policy templates designed to help organizations quickly create and adopt AI governance policies. Built by a digital assets and IP law firm, these templates provide legally-informed starting points for businesses navigating AI compliance. Free to use and adapt without copyright restrictions.
postx.com
CMU AI Agents Course
A university course at Carnegie Mellon University focused on building AI agents end-to-end. Students gain hands-on experience constructing agentic harnesses, building evaluations, and training LLMs with reinforcement learning. The course balances theory and practice while covering modern frameworks and best practices.
postx.com
A Field Guide to Fable: Finding Your Unknowns
An article by Thariq exploring how to work effectively with Claude's Fable system, drawing on the insight that 'the map is not the territory.' It guides readers on understanding the gap between their prompts/context and the actual work being done by AI. Useful for developers and creators trying to get more out of AI-assisted workflows.
postx.com
The Hitchhiker's Guide to Agentic AI
A comprehensive practitioner's reference book hosted on arXiv covering the full stack of building autonomous AI systems, from foundational principles to production deployment. It's positioned as a deep, substantive alternative to the shallow 'edu-slop' threads and quick-read content proliferating on social media. Ideal for engineers and researchers who want to seriously study agentic AI architecture.
postx.com
WTF Is a Loop? Part 2: The 15 Loops People Are Actually Running
A viral article by Matt Van Horn breaking down the 15 most common AI agent loops people are actually running, complete with copyable commands. It's the sequel to a 3.6M-view piece on what loops are, now focused on practical implementation.
postx.com
Reflecting on a Year of Claude Code (Boris Cherny)
Claude Code is an agentic coding tool that runs in your terminal, letting developers delegate coding tasks, verification, code review, and CI automation to an AI agent. It supports advanced workflows like auto mode, agent orchestration, voice control, and remote operation across entire codebases.
videowww.youtube.com
Anthropic's Boris Cherny: Why Coding Is Solved, and What Comes Next
Claude Code is Anthropic's AI coding agent that lets developers ship code through natural language instead of writing it by hand. It supports agentic workflows where you can dispatch multiple PRs at once, even from a phone.
videowww.youtube.com
Boris Cherny: Claude Code & the Future of Engineering | Acquired Unplugged
Claude Code is Anthropic's AI coding agent that lets developers build software through natural language rather than a traditional IDE. It can autonomously write, edit, and reason about code — even building entire applications like Cowork in 8 days.
videowww.youtube.com
Andrej Karpathy on Vibe Coding & Agentic Engineering (AI Ascent 2026)
A conversation between Andrej Karpathy and Sequoia's Stephanie Zhan about the evolution from 'vibe coding' to agentic engineering, the nature of LLMs as 'ghosts,' Software 3.0, and why you can outsource thinking but never understanding.
videowww.youtube.com
Printing Press
Printing Press turns any website or app into a command-line interface that AI agents can use, discovering hidden APIs and integrations so agents can interact with services that lack official developer access. It's built to make agentic workflows work with tools that weren't designed for them.
videowww.youtube.com
Why LLM Inference Is Memory-Bound (Julia Turc)
An educational video breaking down why GPUs vastly underperform their theoretical token throughput during LLM inference. It uses the roofline model to explain memory bandwidth bottlenecks, KV caching, speculative decoding, and how diffusion-based LLMs shift the workload from memory-bound to compute-bound.
videowww.youtube.com
Diode
Diode is an AI-powered platform that designs and manufactures printed circuit boards (PCBs) in the United States, using models that can one-shot much of schematic design and compress hardware timelines from months to weeks. It aims to make hardware development as nimble as software.
videowww.youtube.com
AI Memory Management Philosophy
An article/post arguing that AI agents should store learned lessons as pull requests to shared skill files rather than private memory files. The core insight is that private memory accumulates noise and makes agents dumber over time, while code-reviewed skill files keep agents sharp and auditable. The author advocates limiting memory to 6 files as a forcing function for quality.
postx.com
Rhys's MCP Retrospective
A thread/article by Rhys Sullivan analyzing why MCP (Model Context Protocol) didn't gain traction when it launched alongside early models like Sonnet 3.5 and GPT-4o, and what comes next for agent-to-tool connectivity. Written by a YC-backed founder building infrastructure to connect AI agents to tools, APIs, and services.
postx.com
The Road from 2 GPUs to a 32-GPU Cluster
A first-person narrative article documenting the journey of scaling a personal GPU cluster from a single RTX 3090 to a 32-GPU setup over two years. It covers the practical decisions, challenges, and lessons learned in building serious personal AI/ML infrastructure. Useful for ML practitioners and hobbyists considering scaling their own compute.
postx.com
How to Be Good at Research
A viral thread by Vivek (@itsreallyvivek) that breaks down the unspoken craft of doing good research — the kind nobody explicitly teaches you. It covers how researchers actually learn the job, how to think about novelty, and how to produce meaningful work when the path isn't clear. Useful for academics, independent researchers, and curious professionals alike.
postx.com
Goal + Loss Functions: Distill a Product in 30 Hours Playbook
A detailed playbook by Elvis (@elvissun) explaining how to properly use /goal and loss functions with agentic AI coding tools to build a full product in under 30 hours using a single prompt. It corrects common misconceptions about long-running autonomous agent loops and shares techniques used by top agentic engineers.
postx.com
Index How Vocabulary
A comprehensive reference guide for design and development terminology, covering typography, color theory, and visual design concepts. Provides precise definitions and practical context for designers and developers who need to communicate accurately about visual elements.
postx.com
Just Talk To It - Agentic Engineering Guide
A practical, no-hype guide to working effectively with AI coding agents for software development. Covers real-world workflows, tool comparisons, and best practices for managing multiple agents in parallel.
articlesteipete.me
OpenClaw Orange Book
Complete guide to the OpenClaw open-source AI agent framework, bilingual English and Chinese.
repogithub.com
Claude Constitution
The foundational document describing Claude's values and behavior.
repogithub.com
Codex Orange Book
Complete practical guide to OpenAI Codex — agentic coding workflows, prompts, and patterns for the GPT-5.5 era.
repohuasheng.ai
Harness Engineering Guide
Open guide to building and managing AI agent runtimes — concepts, tutorials, papers, tools, and references.
repogithub.com
Claude Code Orange Book
Claude Code: The Complete Guide — Chinese-language Orange Book series for shipping with AI coding agents.
repogithub.com
OpenAI Model Spec
The OpenAI Model Spec — a public document defining how OpenAI's models should behave, the principles that govern them, and the rules they follow.
repomodel-spec.openai.com
Build nanoGPT
Andrej Karpathy's video lecture + code: build nanoGPT from scratch step by step, training a small GPT on Shakespeare and then OpenWebText.
repogithub.com
promptbase
Microsoft's curated collection of prompt-engineering best practices, techniques, and benchmarks — distilled lessons from running LLMs at scale.
repogithub.com
AI Agents Masterclass
Cole Medin's hands-on AI agents masterclass — video series with companion code covering production-grade agent patterns.
repogithub.com
Neural Networks: Zero to Hero
Andrej Karpathy's free video course building neural networks and language models from scratch in Python — from micrograd through nanoGPT.
repogithub.com
Context Engineering Intro
Cole Medin's hands-on intro to context engineering — the practice that makes AI coding assistants actually work. Claude Code-centric, with templates and patterns.
repogithub.com
Coding LLMs from Scratch
Blaizzy's hands-on series for building LLMs from scratch — code-first walkthroughs of architecture, training, and inference.
repogithub.com
rUv-dev
rUv's AI-powered development methodology — the rUv approach: prompts, patterns, and skills for shipping with AI agents.
repogithub.com
Using Claude Code: The Unreasonable Effectiveness of HTML
Thariq makes the case for switching from Markdown to HTML as the default agent output format. Argues HTML is richer (tables, SVG, CSS, interactivity), easier to share, and keeps you more in-the-loop with what Claude is generating — even though it's 2-4x slower to produce than Markdown.
postx.com
Awesome DESIGN.md
A curated collection of DESIGN.md files inspired by popular brand design systems that AI agents can read to generate consistent, pixel-perfect UI. Simply drop a DESIGN.md file into your project and tell any coding agent to build matching interfaces.
repogithub.com
Prompt caching in LLMs, clearly explained
A case study on how Claude achieves 92% cache hit-rate Every time an AI agent takes a step, it sends the entire conversation history back to the LLM. That includes the system instructions, the tool definitions, and the project context it already processed three turns ago. All of it gets re-read, re-processed, and re-billed on every single turn. For long-running agentic workflows, this redundant computation is often the most expensive line item in your entire AI infrastructure. A system prompt with 20,000 tokens running over 50 turns means 1 million tokens of redundant computation billed at full price, producing zero new value. And that cost compounds across every user and every session. The fix is prompt caching. But to use it well, you need to understand what’s actually happening under the hood. Static vs. Dynamic context Before you can optimize a prompt, you need to understand what changes and what doesn’t. Every agent request has two fundamentally different parts: The static prefix that stays identical across turns: system instructions, tool definitions, project context, and behavioral guidelines. The dynamic suffix that grows with every turn: user messages, assistant responses, tool outputs, and terminal observations. This split is what makes prompt caching possible. The infrastructure stores the mathematical state of the static prefix so that subsequent requests sharing that exact prefix can skip the computation entirely and read from memory. Once you internalize this, every architectural decision in this article becomes obvious. How does the KV Cache work? To understand why caching is so effective, you need to know what the transformer actually does when it processes your prompt. Every LLM inference request has two phases: The prefill phase handles the entire input prompt. It runs dense matrix multiplications across all tokens in context to build the model’s internal representation. This is compute-bound and expensive. The decode phase generates toke
postx.com
Imagine if naked people were stupider. It turns out, naked models actually are.
Sorry, not that kind of naked model. But the disappointment you're feeling right now? That's exactly how Kyle Kingsbury feels about LLMs. Kyle Kingsbury is one of the best systems engineers alive. His Jepsen project spent a decade methodically proving that distributed databases didn't work as advertised. That CockroachDB, MongoDB, Redis, and dozens of others made consistency guarantees they couldn't keep. He published the results, the vendors fixed the bugs, and the entire industry got more honest. Jepsen is a masterwork of applied skepticism. Last week he published a 32-page essay called "The Future of Everything is Lies, I Guess: Bullshit About Bullshit Machines." It's beautifully written, deeply researched, genuinely funny, and wrong about the most important thing. His observations are correct. His conclusion is not. A note on scope: this essay addresses Kingsbury's technical claims: that LLMs are unreliable bullshit machines incapable of producing trustworthy output. His broader concerns about labor displacement, information ecology, and cultural impact are real, separate questions that deserve their own essays. I'm not dismissing them by not addressing them here. I'm addressing the architecture question: does model unreliability make useful systems impossible, or does it make them an engineering problem? I think it's the latter. Kingsbury's essay assumes the former. That's where we disagree. Testing the engine on a bench Kingsbury's essay is structured as a catalogue of LLM failures. He asked Gemini to apply materials to a 3D bathroom rendering. It forgot the toilet and changed the room's shape. He asked Claude to do image-to-image transformation. It produced thousands of lines of JavaScript creating an incomprehensible garble of nonsense polygons. He asked ChatGPT to put white patches on a blue shirt. It changed the color, moved the patches, deleted them. He watched a colleague's LLM claim to download stock data and produce a graph of randomly generated number
postx.com
How to really stop your agents from making the same mistakes
LangChain has raised $160 million. Three years of development. A billion-dollar valuation. LangSmith, their testing platform, is genuinely sophisticated: trajectory evals, trace-to-dataset pipelines, LLM-as-judge, regression suites, unit test frameworks for tools. They have the pieces. Credit where it's due. But pieces aren't a practice. LangChain gives you testing tools. It never tells you what to test, in what order, or when you're done. There's no opinionated workflow that says, in order: this failure happened now write a skill now write the deterministic code now write unit tests now write LLM evals now add a resolver trigger now eval the resolver now audit for duplicates now smoke test now file correctly That loop doesn't exist. You have to invent it yourself from scattered primitives. $160 million in funding, and most LangChain users still don't test their agents, because the framework gave them a gym membership without a workout plan. Most AI agent "reliability" is vibes-based. Prompt tweaks. Bigger system messages. "Please don't hallucinate" incantations. That stuff decays the moment the conversation gets complex. The frameworks that raised hundreds of millions of dollars to solve this gave you monitoring dashboards and unit test helpers and said "good luck." My agent screwed up twice this week. Neither failure can happen again. Not because I asked nicely. Because I turned each failure into a permanent structural fix: a skill with tests that run every day, forever. What hundreds of millions of dollars of VC capital couldn't buy you, I am going to give it to you today for free in open source. I call the practice "skillify." Once you use it, your agents won't keep making the same mistakes. Here's how it works. Failure 1: The Trip That Was Already in the Database I asked my OpenClaw about an old business trip, nearly ten years back, buried somewhere in calendar history. Simple question. Should take one second. Instead the agent did this: Called the live calend
postx.com
The Harness Is Everything: What Cursor, Claude Code, and Perplexity Actually Built
You are not using AI wrong because you haven't found the right model. You are using AI wrong because you haven't built the right environment. There is a reason some teams are shipping a million lines of code with three engineers while others are struggling to get a consistent refactor out of their agent pipeline. The difference is not GPT-5 versus Claude Opus. The difference is not the temperature setting or the max tokens. It isn't even the prompt, though everyone loses months of their life arguing about prompts. The difference is the harness. This article is about what that word actually means, technically and philosophically, because the industry has developed a bad habit of using it loosely. A harness is not a system prompt. It is not a wrapper around an API call. It is not an eval framework or a prompt template or a chatbot with memory. A harness is the complete designed environment inside which a language model operates, including the tools it can call, the format of information it receives, how its history is compressed and managed, the guardrails that catch its mistakes before they cascade, and the scaffolding that allows it to hand off work to its future self without losing coherence. When you look at what Anthropic built to make Claude Code actually work, what OpenAI built to ship a million lines of code through Codex with zero manually-written code, and what the Princeton NLP group published in their landmark SWE-agent paper about agent-computer interfaces, you start to see the same pattern emerging from every serious team working in this space. The model is almost irrelevant. The harness is everything. This is a detailed technical breakdown of how that idea became the defining insight of applied AI engineering in 2025 and 2026. It covers the research, the real implementations, the failure modes that motivated the design decisions, and the patterns that repeat whether you are building a coding agent, a research agent, or a long-running autonomous software
postx.com
AI Agent Stack Everyone Must Use in 2026 (Builder’s Guide)
here is the truth nobody tells ai builders. you do not need to build your own model. all you need to build is TLDR; if you don't wanna read it , then give this link to your agent and ask it questions to understand ➡️github.com/codejunkie99/agentic-stack the infrastructure around the model. The memory that persists across sessions. The skills that encode how tasks should be done. The protocols that govern what the agent can and cannot do. Garry Tan posted on April 13 that nailed this point perfectly. I spent the last three months building exactly this. Four-layer memory. Fat skills with self-rewrite hooks. Protocol enforcement. A nightly dream cycle that compresses everything the agent learned that day. This article is 4,000 words of how I built it and why every piece exists. If you do not want to read all that, you can just install the whole thing here:⬇️ If you just want to install it: github.com/codejunkie99/agentic-stack Works with Claude Code, Cursor, OpenClaw, or any agent that reads markdown. If you want to understand what you are installing and why each piece exists, keep reading.⬇️ The shape of the full stack Here is the folder structure. I will explain each part in detail, but I want you to see the whole thing first because the shape matters more than any individual piece. This mirrors patterns from Gstack and Claude Code's own memory hooks. I did not design it from scratch. I arrived at it by trying other layouts that did not work and gradually converging on this one. The key insight, and the thing Garry's post crystallized for me, is the separation. The harness does not think. It reads files, calls tools, writes logs, runs hooks. All the intelligence lives in the skill files and the memory files. The protocols enforce boundaries. This means: You can swap the harness for a different one tomorrow and lose nothing You can swap the model and lose nothing The only things that accumulate value are the skills, the memory, and the p
postx.com
Introducing EmDash — the spiritual successor to WordPress that solves plugin security
Cloudflare's announcement of EmDash, a new serverless CMS built as a modern WordPress successor. The article explains how EmDash solves WordPress's plugin security issues by running plugins in isolated Workers with explicit permissions.
blogblog.cloudflare.com
Self-Improving Agentic Systems with Auto-Evals
A comprehensive guide and methodology for building AI agents that automatically improve themselves by mining failures, creating evaluations, and fixing issues autonomously. Includes open-source auto-harness library implementation.
postx.com
Notes from the SaaS Funeral
A thought-provoking analysis of why SaaS isn't dead despite AI's ability to generate code, arguing that software businesses require ongoing maintenance, compliance, and AI-native evolution rather than one-time code generation.
postx.com
Design Agent Harness
A three-layer system for engineers to build professional UI/UX using AI agents without becoming designers. Combines design skills, agent canvases, and taste development tools.
postx.com
Open Agentic Commerce
A platform advocating for open protocols in AI-powered commerce, enabling checkout and transactions through AI agents like ChatGPT without centralized gatekeepers. Aims to create an open alternative to walled-garden commerce platforms.
postx.com
Claude + Obsidian Second Brain Setup
A personal knowledge management system that combines Claude AI with Obsidian to create an automated second brain that ingests, processes, and connects all your notes, articles, and ideas without manual maintenance.
postx.com
Resolvers: The Routing Table for Intelligence
In "Thin Harness, Fat Skills", I introduced five definitions for building agent systems that actually work. Skills got all the attention. People bookmarked the skill-as-method-call pattern, the diarization concept, the thin harness architecture. Good. Those matter. But the one that got almost no attention is the one that matters most. Resolvers. And the reason they got ignored is the same reason they're so important: they're invisible when they work, and catastrophic when they don't. A resolver is a routing table for context. When task type X appears, load document Y first. That's it. One sentence. But that one sentence is the difference between an agent that compounds intelligence and an agent that slowly forgets what it knows. This is the story of how I learned that the hard way. The 20,000-line confession My CLAUDE.md was 20,000 lines. I'm not proud of this. Every quirk, every pattern, every lesson I'd ever encountered with Claude Code, every convention for my codebase, every edge case I'd been burned by. I kept adding. The file kept growing. It felt productive. It felt like I was making the model smarter. I wasn't. I was drowning it. The model's attention degraded. Responses got slower and less precise. Claude Code literally told me to cut it back. That's when you know you've gone too far — the AI is telling you to stop talking. The instinct is natural. You want the model to know everything. So you cram everything into the system prompt, the instructions file, the context window. You're trying to make the model omniscient by proximity. It doesn't work. You can't make someone smarter by shouting louder. You make them smarter by giving them the right book at the right moment. The fix was about 200 lines. A numbered decision tree. Pointers to documents. When the model needs to file something, it walks the tree: Is it a person? → /people/ directory A company? → /companies/ directory A policy analysis? → /civic/ directory Twenty thousand lines of knowledge, accessib
postx.com
Meta-Harness: End-to-End Optimization of Model Harnesses
A research paper and framework demonstrating automated optimization of model harnesses - the scaffolding code that determines what AI models store, retrieve, and present during task execution. Shows significant improvements over hand-designed systems across text classification, math reasoning, and coding tasks.
repoyoonholee.com
Using Claude Code: Session Management & 1M Context
Today we’re rolling out a new update to /usage to help you understand your usage with Claude Code, this was informed by a number of conversations with customers. What came up again and again in these calls is that there is a lot of variance in how you might manage your sessions, especially with our new update to 1 million context in Claude Code. Do you only use one session or two sessions that you keep open in a terminal? Do you start a new session with every prompt? When do you use compact, rewind or subagents? What causes a bad compact? There’s a surprising amount of detail here that can really shape your experience with Claude Code and almost all of it comes from managing your context window. A Quick Primer on Context, Compaction & Context Rot The context window is everything the model can "see" at once when generating its next response. It includes your system prompt, the conversation so far, every tool call and its output, and every file that's been read. Claude Code has a context window of one million tokens. Unfortunately using context has a slight cost, which is often called context rot. Context rot is the observation that model performance degrades as context grows because attention gets spread across more tokens, and older, irrelevant content starts to distract from the current task. Context windows are a hard cutoff, so when you’re nearing the end of the context window, you will need to summarize the task you’ve been working on into a smaller description and continue the work in a new context window, we call this compaction. You can also trigger compaction yourself. Every Turn Is a Branching Point Say you've just asked Claude to do something and it's finished, you’ve now got some information in your context (tool calls, tool outputs, your instructions) and you have a surprising number of options for what to do next: Continue — send another message in the same session /rewind (esc esc) — jump back to a previous message and try again from there /clear —
postx.com
How GBrain Works, and How to Actually Wire It Into Your Agents
@garrytan open-sourced his production AI infrastructure: 17,888 pages, 4,383 people dossiers, 723 companies. Here's the architecture. Garry Tan (President & CEO @ycombinator) made public 17,888 pages of his brain. His production system consists of 4,383 people dossiers, 723 companies, 21 autonomous cron jobs created within a span of 12 days. In the first 24 hours this was able to collect over 5,400+ GitHub stars. It’s a personal knowledge layer for AI agents. By default, all agents are stateless, that is each session begins blank. GBrain provides a structured, searchable long-term memory for each session using an embedded Postgres database and a git-tracked Markdown corpus. The agent uses the brain prior to answering questions and updates the knowledge base with new information following each session. The knowledge grows exponentially. MIT licensed. TypeScript, runs on Bun. v0.10.1. Repo snapshot Three-layer architecture Brain Repo: A git-backed directory of Markdown files, one per person, company, project, or idea. The repo is the system of record. Edit any file manually and gbrain sync picks up the changes. Tan's own instance includes 13 years of calendar data, 280+ meeting transcripts, and 300+ captured ideas. Retrieval Layer: Postgres with pgvector. The CLI and MCP server are both generated from a single contract-first interface (src/core/operations.ts, ~30 shared operations). Two pluggable engines: PGLite (embedded Postgres 17.5 via WASM, zero config, boots in 2 seconds) is the default. For corpora over 1,000 files, gbrain migrate --to supabase moves everything to a managed Postgres instance. Migration is bidirectional. Skills Layer: 25 Markdown skill files organized by RESOLVER.md. The architecture philosophy is "thin harness, fat skills": executable TypeScript handles deterministic operations (search, embed, import, sync). The agent model handles judgment-dependent tasks by reading the skill file and executing it. Skills are plain text and modifiabl
postx.com
AutoAgent: Self-Optimizing AI Agents
An open source library that uses a meta-agent to autonomously improve task agents by experimenting with prompts, tools, and orchestration until performance climbs. Achieved #1 scores on SpreadsheetBench and TerminalBench through fully automated optimization over 24+ hours.
postx.com
What Happened When I Applied Karpathy's Autoresearch Idea to LLM Inference
Most "AI optimization" demos are fun to watch for the same reason benchmark tweets are fun to watch: they show you the win, not the search. You see the final graph. You see the +12% or the "runs 2x faster now" claim. What you usually do not see is the graveyard of bad ideas behind it. The settings that looked promising but were just noise. The optimizations that made throughput better by quietly making the model worse. The fake wins that only happened because the benchmark got easier. So I built a small repo called Auto-Inference-Optimiser (star the repository!) to study exactly that. The idea is simple: lock the evaluation, open one file for experimentation, and let an AI coding agent hill-climb on inference speed forever on Apple Silicon. The most interesting part was not that it achieved a speedup. It was the kind of speedup it achieved, what it failed to improve, and what that indicates about inference engineering on real hardware. Let's get into it. Why I Built This I care a lot about inference right now. Not in the abstract "LLMs are cool" sense. I mean the actual production questions: where latency comes from, what batching buys you, what prompt processing costs, how KV cache decisions change throughput, and where the hardware wall starts pushing back. There is a lot of content online about training. There is also a lot of content online about agents. But there is still not enough content that combines both instincts: build a tight experimental harness, let the agent search inside it, and use that process to learn something real about inference. This repo was my way of doing that. It is inspired by Karpathy's Autoresearch, but pointed at a different layer of the stack. Instead of searching over training code on a GPU box, this one searches over an MLX inference pipeline on a Mac because I am GPU poor (please sponsor a GPU). What The Repo Actually Does At a high level, the repo turns "make inference faster" into a bounded optimization problem. The structure is
postx.com
.claude Folder Configuration Guide
A comprehensive guide explaining how to configure Claude Code's .claude folder, including CLAUDE.md instructions, custom commands, permissions, and project setup. Helps developers optimize their AI coding assistant for team workflows.
postx.com
The Input Layer
An educational article teaching advanced prompt engineering techniques for AI models, emphasizing detailed 2000+ word prompts over simple instructions. Written by an AI-native law firm managing partner sharing professional-grade prompting strategies.
postx.com
Building CLIs for AI Agents
A comprehensive guide on how to design command-line interfaces that work effectively with AI agents. Provides practical patterns for making CLIs non-interactive, agent-friendly, and robust for automated use.
postx.com
Claude Setup Detox Audit
A self-audit prompt system that helps Claude users identify and remove redundant or conflicting instructions from their custom setups. Uses Claude's ability to analyze its own configuration files to recommend what to cut for better outputs.
postx.com
Deep Agents Evaluation Framework · article
An X post walking through how to evaluate deep agents — the author's approach to curating eval data, defining metrics, and running experiments to measure whether agent changes actually help. A write-up of an approach, not a package you install.
postx.com
How to Become So Creative It Feels Illegal
A creative thinking guide that offers a 7-day protocol to overcome mental blocks and unlock heightened creativity. Explores the cognitive barriers that limit creative thinking and provides practical steps to return to a natural creative state.
postx.com
From Hierarchy to Intelligence
An in-depth analysis of how organizational structures evolved from Roman military hierarchies to modern corporations, exploring how AI might fundamentally transform how teams work together beyond just productivity gains.
postx.com
Claude Code Best Practices Guide
A comprehensive guide and reference for Claude Code development, covering subagents, commands, skills, workflows, and advanced features. Provides structured best practices, implementation examples, and educational content for building AI-powered development workflows.
repogithub.com
Services: The New Software
A strategic analysis by Sequoia Capital exploring how AI-powered service companies will capture more value than traditional software tools by selling outcomes instead of features. Makes the case that the next $1T company will be disguised as a services firm.
articlesequoiacap.com
Claude Token Optimization Guide
A comprehensive guide with 9 practical strategies to reduce Claude token usage and avoid hitting usage limits. Covers prompt editing, batching questions, using projects, and choosing the right models to save money.
postx.com
Legal AI Analysis
An analysis of why current AI tools for lawyers fail on real legal matters and how map-reduce reasoning offers a better approach than traditional RAG systems. Explains the technical limitations of chunking legal documents and proposes structured document indexing.
postx.com
Agent Harness Memory Control · article
An X post arguing that who controls your agent's memory is inseparable from the harness, and that open harnesses avoid vendor lock-in. A strategic argument to read, not a tool or library.
postx.com
The SaaS Reckoning
An analysis piece examining how stock-based compensation has hidden true costs in SaaS companies and how AI disruption is forcing a reckoning with these accounting practices. Explores the financial reality behind adjusted EBITDA metrics and their impact on company valuations.
postx.com
AI Autoresearch Strategy Generator
An AI-powered system that uses parallel Claude agents to automatically generate, test, and optimize trading strategies. It won a prediction market hackathon by creating 1,039 unique strategy variants without human domain expertise.
postx.com
Thin Harness, Fat Skills · article
A short essay arguing that productive AI coding agents come from a lightweight harness plus composable skills. Explains the context-management and workflow ideas behind the '10-100x' claim. An opinion piece, not a harness you can run.
postx.com
Agent Memory Architecture Guide
A comprehensive technical guide explaining how to implement memory systems for AI agents, covering everything from basic Python lists to advanced graph-vector hybrid approaches. Provides practical solutions for building agents that can remember and learn from past interactions.
postx.com
ChatDev 2.0
A zero-code multi-agent orchestration platform that lets users build and execute customized multi-agent systems through simple configuration. Users can define agents, workflows, and tasks to orchestrate complex scenarios like data visualization, 3D generation, and deep research without any coding required.
repoarxiv.org
TradingAgents
Multi-agent LLM framework that simulates real-world trading firms by deploying specialized AI agents for fundamental analysis, sentiment analysis, technical analysis, and risk management. The agents collaborate and debate to make informed trading decisions through structured discussions.
repoarxiv.org
Curated by a proprietary index. Corrections welcome.