Intel
Page 50
We're Cutting L40S Prices In Half
Fly halves L40S GPU pricing to $1.25/hour and explains the demand picture behind it.
Why it mattersL40S GPU hours drop to $1.25, and the provider's own demand data shows cheaper A10s dominate because they handle mid-sized generative workloads like Mistral Nemo and Stable Diffusion well enough.
articlefly.io
articleQwen Team
Qwen2-Audio: Chat with Your Voice!
Qwen2-Audio extends the Qwen family to audio.
Why it mattersQwen2-Audio is an open multimodal model that natively accepts audio and text and returns text, enabling voice chat and audio analysis without stitching together a separate speech-to-text pipeline.
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Showed compute spent at inference can outperform compute spent on a larger model, and that how best to spend it shifts with the difficulty of the prompt.
Why it mattersThe result behind reasoning models: for many problems letting a smaller model think longer beats training a larger one, and the optimal strategy shifts with prompt difficulty.
articleCharlie Snell et al.
An Open Course on LLMs, Led by Practitioners
Mastering LLMs, an open course of workshops and talks from 25-plus practitioners covering evals, retrieval-augmented generation and fine-tuning.
Why it mattersA free, well-organized 40+ hour course distilled from a popular paid program, with annotated talks and notes from practitioners across evals, RAG, and fine-tuning — a fast way to level up on shipping real LLM products rather than toy demos.
articleHamel Husain
Building A Generative AI Platform
The components that recur across generative AI platforms once you look at how companies actually deploy them, built up from the simplest possible architecture rather than presented as a finished diagram.
Why it mattersA clear, incrementally-built reference architecture for production genAI systems — showing when and why to add RAG, guardrails, model gateways, caching, and orchestration.
articleChip Huyen
Extrinsic Hallucinations in LLMs
Narrowing hallucination to its useful meaning: output that is fabricated and grounded in neither the provided context nor world knowledge, rather than any mistake a model makes.
Why it mattersIt gives a precise taxonomy of hallucination (in-context vs. extrinsic) and surveys the actual detection and mitigation methods.
articleLilian Weng
AI Engineer 2024 Keynote - What We Learned from a Year of LLMs
Special double-feature closing keynote from the 6 authors of the hit O'Reilly article on Applied LLMs.
Why it mattersA concentrated set of production LLM lessons from six practitioners covering evals, prompting, RAG vs. fine-tuning tradeoffs, and operational pitfalls—useful if you're moving an LLM feature from demo to reliable product.
articleEugene Yan
Introducing Gen 3 Alpha
Runway's Gen-3 Alpha, the first model on their new multimodal training infrastructure, improves fidelity, motion and photorealistic human generation over Gen-2 and powers text-to-video, image-to-video and control modes including motion brush and camera direction.
Why it mattersTraining on temporally dense captions gives fine-grained control over when things happen in a shot, enabling keyframed transitions and expressive human performance the prior generation could not sustain.
articleRunway editorial sitemap
articleQwen Team
Generalizing an LLM from 8k to 1M Context using Qwen-Agent
Qwen builds an agent from 8k-context Qwen2 models that understands million-token documents, beating both RAG and native long-context models, then uses it to generate training data for new long-context Qwen models.
Why it mattersShows how to make a small-context model comprehend million-token documents by building a chunking-and-reasoning agent rather than relying on RAG or expensive long-context models.
Netflix PRS 2024 - Applying LLMs to Recommendation Experiences
Challenges and lessons from deploying LLM experiences: evals, scalability, guardrails.
Why it mattersA real-world postmortem from Netflix on shipping LLMs into recommendation experiences at scale, with practical guidance on building evals, handling scalability, and designing guardrails that most tutorials skip.
articleEugene Yan
articleEugene Yan
Prompting Fundamentals and How to Apply them Effectively
Structured input/output, prefilling, n-shots prompting, chain-of-thought, reducing hallucinations, etc.
Why it mattersThe prompting techniques that survive contact with production: structured I/O, prefilling, n-shot selection and chain-of-thought, with when-to-use guidance.
What We've Learned From A Year of Building with LLMs
From the tactical nuts & bolts to the operational day-to-day to the long-term business strategy.
Why it mattersA dense, practitioner-written distillation of what actually works when shipping LLM apps — covering prompting patterns, RAG, evaluation strategy, and the operational/organizational realities most tutorials skip.
articleEugene Yan
Picture This: Open Source AI for Image Description
A blind Fly engineer builds image description on open-source vision models, using everyday accessibility problems — find the coffee cup, find the light switch.
Why it mattersA worked example of assembling open-source vision-language models into an accessibility tool, showing what self-hosted image description can do versus commercial apps like Be My AI or Seeing AI.
articlefly.io
Debugging AI With Adversarial Validation
Adversarial validation as a simple way to detect drift.
Why it mattersA cheap, infrastructure-free way to catch the silent killer of eval validity — your eval set no longer matching what production actually sends your model.
articleHamel Husain
Diffusion Models for Video Generation
Diffusion models moved from image synthesis to video, a strictly harder problem because temporal consistency across frames demands world knowledge a single image never needed.
Why it mattersA structured survey of how diffusion models extend from image to video generation, unpacking the temporal-consistency and data-scarcity problems that define the current research frontier.
articleLilian Weng
Task-Specific LLM Evals that Do & Don't Work
Evals for classification, summarization, translation, copyright regurgitation, and toxicity.
Why it mattersIt breaks down which eval approaches actually work for specific tasks like summarization, translation, and toxicity detection.
articleEugene Yan
Your AI Product Needs Evals
Hamel Husain's argument that unsuccessful LLM products almost always share one root cause, the absence of a robust evaluation system, drawn from five years of building and consulting on them.
Why it mattersThis is the reference playbook for eval systems — the single highest-leverage practice separating LLM products that improve past the demo stage from ones that stall.
articleHamel Husain
articleQwen Team
Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters
Qwen1.5-MoE-A2.7B is a small mixture-of-experts model that matches 7B performance with about a third of the parameters activated, part of the wave of MoE work that followed Mixtral.
Why it mattersQwen1.5-MoE-A2.7B delivers roughly 7B-class quality while only activating 2.7B parameters, meaning cheaper and faster inference for anyone self-hosting or running cost-sensitive LLM workloads.
articleHamel Husain
Is Fine-Tuning Still Valuable?
Hamel Husain's answer to growing disillusionment with fine-tuning: it remains valuable in specific situations, and the case against it usually generalises from the wrong ones.
Why it mattersGives a clear decision framework for whether to invest in fine-tuning.
What I learned from looking at 900 most popular open source AI tools
What a survey of the 900 most popular open-source AI repositories shows about how the ecosystem has changed, four years after the same exercise on the open ML landscape.
Why it mattersA structured, data-driven map of the open source AI tooling landscape — organized into infrastructure, model development, and application layers.
articleChip Huyen
Predictive Human Preference: From Model Ranking to Model Routing
Instead of ranking models and picking one, predict which model a user would prefer for a specific prompt, turning human preference data from a leaderboard into a routing signal.
Why it mattersIf you're deciding which LLM to call for each prompt, this lays out how to predict which model a user will prefer per-query and route accordingly.
articleChip Huyen
Don't Mock Machine Learning Models In Unit Tests
How unit testing machine learning code differs from typical software practices
Why it mattersIf you write tests for ML pipelines, this explains why mocking models hides real failures and what to validate instead — practical guidance for keeping AI code reliable as it evolves.
articleEugene Yan
How to Generate and Use Synthetic Data for Finetuning
Overcoming the bottleneck of human annotations in instruction-tuning, preference-tuning, and pretraining.
Why it mattersIf you're finetuning models and blocked by the cost of human annotation, this walks through concrete synthetic data generation strategies for instruction-tuning, preference-tuning, and pretraining.
articleEugene Yan
Thinking about High-Quality Human Data
Most task-specific labelled data comes from human annotation, which makes annotator disagreement, expertise and incentives a modelling problem rather than an operational detail.
Why it mattersA deep dive into the mechanics of high-quality human annotation and RLHF labeling — covering rater agreement, aggregation, and quality-control techniques that directly affect the data your alignment and fine-tuning pipelines depend on.
articleLilian Weng
Generation configurations: temperature, top-k, top-p, and test time compute
How generation is actually configured.
Why it mattersIt demystifies why LLM outputs are inconsistent and how sampling knobs (temperature, top-k, top-p) plus test-time compute actually shape responses.
articleChip Huyen
Out-of-Domain Finetuning to Bootstrap Hallucination Detection
How to use open-source, permissive-use data and collect less labeled samples for our tasks.
Why it mattersIf you're building hallucination detection or other classifiers with limited labeled data, this shows how to bootstrap from out-of-domain, permissively-licensed datasets to cut annotation costs while still hitting task performance.
articleEugene Yan
Adversarial Attacks on LLMs
A survey of adversarial attacks and jailbreak prompts against language models.
Why it mattersA rigorous, research-grounded map of how jailbreaks and adversarial prompts actually work against aligned LLMs, giving engineers the vocabulary and threat models needed to red-team and harden their own AI products.
articleLilian Weng
Mitigating Stereotypical Biases In Text To Image Generative Systems
Runway's diversity fine-tuning approach attacks text-to-image skew at the data level.
Why it mattersThe approach offers a concrete recipe for debiasing text-to-image models using synthetic data built from combinatorial prompt attributes, with released prompts and code and measured group fairness gains on both SD1.5 and SDXL.
articleRunway editorial sitemap
Multimodality and Large Multimodal Models (LMMs)
A primer on multimodality and large multimodal models.
Why it mattersA clear, foundational-to-frontier walkthrough of how multimodal models actually work — from CLIP and Flamingo to modern adapter-based LMMs — giving engineers the conceptual grounding to reason about and build with vision-language systems.
articleChip Huyen
Scale Speed And Stepping Stones The Path To Gen 2
Runway's CTO walks through the sequencing behind the Gen models.
Why it mattersThis explains why video generation was staged, with structure-conditioned Gen-1 coming before open text-to-video, and frames next-frame prediction as the video analogue of next-token prediction for learning world structure.
articleRunway editorial sitemap
Efficient Memory Management for Large Language Model Serving with PagedAttention
Applies operating-system paging to the KV cache so serving stops wasting most of its memory on fragmentation, raising throughput several-fold.
Why it mattersThe reason an inference server can hold many concurrent requests: treating the KV cache like virtual memory pages instead of one contiguous block removes the fragmentation that was wasting most of the GPU.
articleWoosuk Kwon et al.
Evaluation & Hallucination Detection for Abstractive Summaries
Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.
Why it mattersIf you're shipping summarization features, this lays out concrete metrics — reference-based, context-based, preference-based, and self-consistency checks.
articleEugene Yan
Open challenges in LLM research
Ten major open research directions in language models, drawn from conversations across industry and academia.
Why it mattersA structured tour of the 10 hardest open problems in LLM research — from measuring hallucination to non-GPU compute — with curated primary papers for each, giving engineers a map of where the field is heading and what to watch.
articleChip Huyen
Sidekicks Improved Streaming
Shopify's Sidekick team details two fixes for streaming chat UX.
Why it mattersStreaming LLM output re-renders half-formed Markdown as raw text and stalls behind multi-hop tool calls; selective buffering plus a multiplexed event stream fixes both without waiting for the full response.
articleShopify Engineering
Patterns for Building LLM-based Systems & Products
Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.
Why it mattersA comprehensive field guide to the seven practical patterns behind production LLM systems—covering how to structure evals, RAG, caching, and guardrails—so you can build reliable AI products instead of stitching together demos.
articleEugene Yan
LLM Powered Autonomous Agents
A survey of LLM-powered autonomous agents built around planning, memory and tool use, drawing on AutoGPT, GPT-Engineer and BabyAGI as early proofs of concept.
Why it mattersA clear, canonical breakdown of the core components of LLM agents — planning with subgoal decomposition and reflection, short/long-term memory via vector stores, and tool use.
articleLilian Weng
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Derives preference training as a simple classification loss on the policy itself, removing the reward model and the reinforcement-learning loop from RLHF.
Why it mattersIt stripped most of the machinery out of preference training: no separate reward model, no RL loop, just a classification loss that reaches comparable quality and is stable enough to run without a research team.
articleRafael Rafailov et al.
QLoRA: Efficient Finetuning of Quantized LLMs
Backpropagates through a frozen 4-bit quantized model into LoRA adapters, putting 65B-parameter finetuning on a single GPU without measurable quality loss.
Why it mattersIt collapsed the hardware bar for fine-tuning a large model from a cluster to one GPU, by backpropagating through 4-bit frozen weights into low-rank adapters.
articleTim Dettmers et al.
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Interpolates between multi-head and multi-query attention using grouped key-value heads, recovering multi-query speed while keeping most of multi-head quality.
Why it mattersThe compromise nearly every current model ships: grouped key-value heads give you multi-query's small cache without the quality drop, and the paper shows you can convert an existing multi-head checkpoint rather than retrain from scratch.
articleJoshua Ainslie et al.
Some Intuition on Attention and the Transformer
What's the big deal, intuition on query-key-value vectors, multiple heads, multiple layers, and more.
Why it mattersA clear, intuition-first walkthrough of query-key-value attention, multi-head, and multi-layer mechanics — useful if you want to actually understand why the Transformer works rather than just memorize the diagram.
articleEugene Yan
Minimal Chat
An argument that chat became the default LLM interface by inertia rather than fit.
Why it mattersChallenges the reflexive choice of a chat box as the LLM interface, arguing better interaction patterns exist — a design decision that shapes every AI product before a single model call is made.
articleeugeneyan.com
More Design Patterns For Machine Learning Systems
9 patterns including HITL, hard mining, reframing, cascade, data flywheel, business rules layer, and more.
Why it mattersA compact catalog of production ML design patterns — like cascade models, hard negative mining, data flywheels, and business-rules layers.
articleEugene Yan
Prompt Engineering
A survey of prompt engineering as in-context steering of an autoregressive model without touching its weights, treated as an empirical science whose methods vary sharply between models.
Why it mattersA rigorous, practitioner-oriented survey of in-context prompting methods — few-shot, chain-of-thought, self-consistency, and more.
articleLilian Weng
Gen 2
Runway's second-generation video model dropped the requirement for structure conditioning: text alone, or a single driving image, produces a clip.
Why it mattersGen-2 was the point at which text-to-video became usable without a driving video, establishing the mode vocabulary covering text, image, stylization, mask, and render that later video models inherited.
articleRunway editorial sitemap
Shopifys Machine Learning Platform Real Time Predictions
How Shopify extended its Ray-based ML platform from batch to real-time prediction.
Why it mattersShows how a general-purpose model-serving layer is built on Ray with a Feast-backed offline/online feature store, and where the latency and generalization tradeoffs land when many teams deploy to one platform.
articleShopify Engineering
The Transformer Family Version 2.0
A rewritten and roughly doubled survey of transformer architecture variants, restructured and brought up to date with the improvements proposed since the original 2020 version.
Why it mattersA thorough, notation-consistent reference on transformer architecture variants — from attention mechanisms to positional encodings and efficiency tricks.
articleLilian Weng
Large Transformer Model Inference Optimization
Why inference on large transformers is hard and what actually helps.
Why it mattersA rigorous survey of how to cut transformer inference cost in time and memory — quantization, pruning, sparsity, and distillation — giving engineers a practical map for deploying large models affordably at scale.
articleLilian Weng
articleQwen Team
OFASys: Enabling Multitask Learning with One Line of Code!
Qwen's OFASys tackles the practical pain of multimodal multitask learning, the batching and stability problems the team hit building OFA, aiming to make multitask setups a one-line affair.
Why it mattersIf you're wrangling multitask, multimodal training pipelines, OFASys reduces the boilerplate of setting up heterogeneous tasks and modalities into a declarative interface, tackling the batchification and training-stability headaches that usually derail generalist-model experiments.
articleQwen Team
Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese
CLIP became both a foundation model and a bridge between vision and language, but cross-modal retrieval needs language-specific versions. This is an open-sourced Chinese CLIP.
Why it mattersIf you're building cross-modal retrieval or text-to-image pipelines for Chinese content, this gives you a purpose-trained Chinese CLIP rather than forcing English-centric embeddings onto Chinese text and image data.
Constitutional AI: Harmlessness from AI Feedback
Anthropic's method for training a harmless assistant from AI feedback against a written set of principles, replacing most human harm labels with model self-critique.
Why it mattersIt is how you train a model to refuse well without a human labelling every bad output: the model critiques and revises its own responses against an explicit written constitution.
articleYuntao Bai et al.
An index of the vibe-coding frontier. Corrections welcome.