Introduces a non-transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → architecture aimed at unbounded context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → with lower compute cost, relevant to anyone tracking long-context limits and inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → economics.
Terms in this piece · Glossary
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
videoWorld Models & General Intuition: Khosla's largest bet since LLMs & OpenAI
videoExtreme Harness Engineering: 1M LOC, 1B toks/day, 0% human code or review — Ryan Lopopolo, OpenAI
videoInside Abridge: The AI Listening to 100 Million Doctor Visits — Abridge's Janie Lee & Chai Asawa
videoExo: Harnesses should see their own code and logs — Alex Krentsel, UC Berekeley / Google Research
