Intel
Page 129
AlignEval: Building an App to Make Evals Easy, Fun, and Automated
Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
Why it mattersIf you're building LLM-as-judge evaluators, this walks through a practical loop for labeling data and optimizing your evaluator against those labels — grounding eval quality in human-aligned measurement rather than vibes.
Introducing Act One
Act One animates generated characters directly from phone-grade video of a performance, preserving eye-lines, micro-expressions, and delivery.
Why it mattersFacial performance transfer from one camera and one actor collapses a mocap-and-rigging pipeline into a single model call, and holds up across characters with proportions unlike the source.
How Shopify Improved Consumer Search Intent With Real Time Ml
How Shopify runs semantic storefront search.
Why it mattersA concrete architecture for keeping embeddings fresh at scale — shared embedding primitives plus streaming inference — which is the hard part of shipping semantic search that batch reindexing quietly hides.
Announcing Our Updated Responsible Scaling Policy
Anthropic's revised risk framework sets capability thresholds that trigger stronger ASL safeguards, adopts safety case methodology for judging whether protections are adequate.
Why it mattersDefines the capability thresholds and matching safeguard standards that determine when Anthropic adds deployment restrictions to a model, which is the machinery behind access limits and safeguards on later frontier releases.
Machines of Loving Grace: How AI Could Transform the World for the Better
An essay by Anthropic CEO Dario Amodei arguing that despite his company's focus on AI risk, powerful AI could bring radical positive transformation across five domains.
Why it mattersOffers a rare detailed, domain-by-domain articulation of AI's upside from a leader whose company is otherwise known for risk-focused messaging, useful context for engineers navigating the discourse shaping frontier AI development priorities.
Foundations For Safe Generative Media
Runway publishes the measured performance of its own visual moderation model against third-party APIs, reporting better F1 and recall at half the false-positive rate, alongside the diversity fine-tuning it uses to keep profession prompts off default demographics.
Why it mattersGives real comparison numbers for in-house visual moderation versus third-party APIs, including the false-positive tradeoff anyone shipping a generative media product has to price in.
Reka Flash Updates
Reka Flash's update adds arbitrary-resolution image handling with stronger OCR and structured output, native audio understanding inside video, 3-5 minute clips instead of one.
Why it mattersInterleaved image, video and audio input in one 21B model at 128K context, with video length up from 1 minute to 3-5 and structured-output support, enough to build video retrieval and segment-summarisation flows on.
AI GPU Clusters, From Your Laptop, With Livebook
How three Elixir pieces — Livebook notebooks, FLAME's elastic executor pools, and the Nx/Axon tensor stack.
Why it mattersShows a concrete path to running GPU ML workloads from a local notebook by marking code with Flame.call and letting a pool of remote executors scale to zero — Elixir-native inference without splitting the app into serverless pieces.
Introducing Contextual Retrieval
Introducing Contextual Retrieval
Why it mattersIf you're building RAG pipelines, Contextual Retrieval shows how prepending chunk-specific context before embedding (and BM25 indexing) cuts retrieval failures substantially over naive chunking.
Qwen2.5: A Party of Foundation Models!
Qwen2.5 arrives as what the team calls possibly the largest open-source release in history, a family of foundation models built on three months of developer feedback since Qwen2.
Why it mattersQwen2.5 is one of the largest open-weight model releases available, spanning many parameter sizes with strong coding and reasoning gains — useful when you need capable, self-hostable alternatives to closed frontier APIs.
Qwen2.5-LLM: Extending the boundary of LLMs
Qwen details the Qwen2.5 language model series.
Why it mattersQwen2.5 gives you a full ladder of open-weight models (0.5B to 72B) with sizes deliberately tuned for production (10-30B) and mobile (3B) deployment.
Qwen2.5-Coder: Code More, Learn More!
Qwen2.5-Coder is the next generation of Qwen's open code models, renaming CodeQwen to Qwen-Coder and building on the CodeQwen1.5 release from earlier that year.
Why it mattersQwen2.5-Coder is a strong open-weight coding model family that can power self-hosted coding agents and IDE tooling without relying on closed APIs, giving engineers a competitive local alternative to GPT/Claude for code generation.
Qwen2.5-Math: The world's leading open-sourced mathematical LLMs
Qwen2.5-Math open-sources 1.5B, 7B and 72B base and instruct models for mathematical reasoning in English and Chinese through chain-of-thought and tool-integrated reasoning.
Why it mattersQwen2.5-Math offers open-weight math-specialized models (1.5B/7B/72B) that combine chain-of-thought and tool-integrated reasoning plus a dedicated reward model.
Introducing The Runway Api
Runway shipped its first public API, exposing the Gen-3 Alpha Turbo video model for integration into third-party products.
Why it mattersGen-3 Alpha Turbo became reachable programmatically rather than only through Runway's own editor, opening generative video as a backend call inside another product. Initial access was partner-gated with a broader opening promised.
Multimodal With Reka Mongodb
A walkthrough of why text-first RAG stalls on charts, tables and PDFs, and why CLIP-style embeddings do not rescue it.
Why it mattersGeneric image embeddings can separate a cat from a dog but not two tables in a financial report; converting charts and tables to markdown before indexing is a workable route to retrieval over multimodal documents.
OpenAI o1 Results on ARC-AGI-Pub
ARC Prize tests OpenAI's o1 models on ARC-AGI-Pub with its standard harness.
Why it mattersShows how test-time reasoning models perform on tasks that need novel adaptation rather than memorized patterns. That helps engineers calibrate what reasoning models can and cannot generalize to.
A New Initiative For Developing Third Party Model Evaluations
Anthropic opened funding for third-party evaluations and, in doing so, mapped where it thinks the eval landscape falls short.
Why it mattersNames the eval categories a frontier lab considers undersupplied, including CTF-style cyber tasks without published solutions and autonomy benchmarks tied to junior through expert research engineer levels, with funding attached.
Projects
Claude Pro and Team gained Projects.
Why it mattersProjects attach a persistent 200K token corpus and custom instructions to a set of conversations, so grounding documents and role framing no longer get re-pasted into every chat.
Updating Our Usage Policy
Anthropic's Acceptable Use Policy became the Usage Policy effective June 6, 2024.
Why it mattersChanges what you may build on the API: high-risk healthcare and legal integrations take on extra safety requirements, products serving minors need disclosure and safeguards, and political campaigning uses are spelled out as prohibited.
Model Safety Bug Bounty
An invite-only HackerOne program aimed at universal jailbreaks, paying up to $15,000, tested against a next-generation safeguard system that has not shipped publicly.
Why it mattersPays up to $15,000 for universal jailbreaks and gives selected researchers access to an unreleased safety mitigation system before public deployment.
Expanding Access To Claude For Government
Anthropic placed Claude 3 Haiku and Sonnet in AWS GovCloud and the AWS Marketplace for the US Intelligence Community, paired with contractual Usage Policy exceptions for legally authorized foreign intelligence analysis.
Why it mattersClaude became deployable in AWS GovCloud and the Intelligence Community marketplace.
Qwen2-VL: To See the World More Clearly
Qwen2-VL is the vision-language release in the Qwen2 family.
Why it mattersQwen2-VL delivers state-of-the-art visual understanding across variable image resolutions and can reason over 20+ minute videos, making it a strong open option for document parsing, visual QA.
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Use cases, techniques, alignment, finetuning, and critiques against LLM-evaluators.
Why it mattersIf you're building LLM-as-Judge evaluators, this breaks down alignment techniques, finetuning approaches, and the concrete failure modes of using LLMs to grade LLMs.
We're Cutting L40S Prices In Half
Fly halves L40S GPU pricing to $1.25/hour and explains the demand picture behind it.
Why it mattersL40S GPU hours drop to $1.25, and the provider's own demand data shows cheaper A10s dominate because they handle mid-sized generative workloads like Mistral Nemo and Stable Diffusion well enough.
Qwen2-Audio: Chat with Your Voice!
Qwen2-Audio extends the Qwen family to audio.
Why it mattersQwen2-Audio is an open multimodal model that natively accepts audio and text and returns text, enabling voice chat and audio analysis without stitching together a separate speech-to-text pipeline.
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Showed compute spent at inference can outperform compute spent on a larger model, and that how best to spend it shifts with the difficulty of the prompt.
Why it mattersThe result behind reasoning models: for many problems letting a smaller model think longer beats training a larger one, and the optimal strategy shifts with prompt difficulty.
An Open Course on LLMs, Led by Practitioners
Mastering LLMs, an open course of workshops and talks from 25-plus practitioners covering evals, retrieval-augmented generation and fine-tuning.
Why it mattersA free, well-organized 40+ hour course distilled from a popular paid program, with annotated talks and notes from practitioners across evals, RAG, and fine-tuning — a fast way to level up on shipping real LLM products rather than toy demos.
Building A Generative AI Platform
The components that recur across generative AI platforms once you look at how companies actually deploy them, built up from the simplest possible architecture rather than presented as a finished diagram.
Why it mattersA clear, incrementally-built reference architecture for production genAI systems — showing when and why to add RAG, guardrails, model gateways, caching, and orchestration.
Extrinsic Hallucinations in LLMs
Narrowing hallucination to its useful meaning: output that is fabricated and grounded in neither the provided context nor world knowledge, rather than any mistake a model makes.
Why it mattersIt gives a precise taxonomy of hallucination (in-context vs. extrinsic) and surveys the actual detection and mitigation methods.
AI Engineer 2024 Keynote - What We Learned from a Year of LLMs
Special double-feature closing keynote from the 6 authors of the hit O'Reilly article on Applied LLMs.
Why it mattersA concentrated set of production LLM lessons from six practitioners covering evals, prompting, RAG vs. fine-tuning tradeoffs, and operational pitfalls—useful if you're moving an LLM feature from demo to reliable product.
Introducing the ARC-AGI Public Leaderboard
ARC Prize introduces ARC-AGI-Pub, a public-eval leaderboard that lifts compute and internet restrictions so frontier APIs can be compared, backed by a $150,000 verification fund.
Why it mattersClarifies the difference between constrained competition scores and unconstrained frontier-model scores on ARC-AGI, which matters when comparing cost and performance claims.
Introducing Gen 3 Alpha
Runway's Gen-3 Alpha, the first model on their new multimodal training infrastructure, improves fidelity, motion and photorealistic human generation over Gen-2 and powers text-to-video, image-to-video and control modes including motion brush and camera direction.
Why it mattersTraining on temporally dense captions gives fine-grained control over when things happen in a shot, enabling keyframed transitions and expressive human performance the prior generation could not sustain.
Voyage AI released a multilingual embedding model reporting an average 5.6% gain across evaluated languages including French, German, Japanese, Spanish and Korean, with a 32K context window and availability through the AWS Marketplace.
Why it mattersA multilingual retrieval option with a 32K context, useful when your corpus is not English and current embeddings degrade on it.
Generalizing an LLM from 8k to 1M Context using Qwen-Agent
Qwen builds an agent from 8k-context Qwen2 models that understands million-token documents, beating both RAG and native long-context models, then uses it to generate training data for new long-context Qwen models.
Why it mattersShows how to make a small-context model comprehend million-token documents by building a chunking-and-reasoning agent rather than relying on RAG or expensive long-context models.
Netflix PRS 2024 - Applying LLMs to Recommendation Experiences
Challenges and lessons from deploying LLM experiences: evals, scalability, guardrails.
Why it mattersA real-world postmortem from Netflix on shipping LLMs into recommendation experiences at scale, with practical guidance on building evals, handling scalability, and designing guardrails that most tutorials skip.
Prompting Fundamentals and How to Apply them Effectively
Structured input/output, prefilling, n-shots prompting, chain-of-thought, reducing hallucinations, etc.
Why it mattersThe prompting techniques that survive contact with production: structured I/O, prefilling, n-shot selection and chain-of-thought, with when-to-use guidance.
What We've Learned From A Year of Building with LLMs
From the tactical nuts & bolts to the operational day-to-day to the long-term business strategy.
Why it mattersA dense, practitioner-written distillation of what actually works when shipping LLM apps — covering prompting patterns, RAG, evaluation strategy, and the operational/organizational realities most tutorials skip.
Picture This: Open Source AI for Image Description
A blind Fly engineer builds image description on open-source vision models, using everyday accessibility problems — find the coffee cup, find the light switch.
Why it mattersA worked example of assembling open-source vision-language models into an accessibility tool, showing what self-hosted image description can do versus commercial apps like Be My AI or Seeing AI.
Voyage AI's instruction-tuned embedding model took the top MTEB position with a 1024-dimension output, four times smaller than any other non-Voyage model in the top five, and a 16K context window, double OpenAI's v3 large.
Why it mattersSmaller vectors at comparable retrieval accuracy cut vector database cost directly, and 16K context lets you embed longer chunks without splitting them.
Vibe Eval
Vibe-Eval opens 269 expert-written image-text prompts with ground-truth answers, built hard enough that frontier models miss half the hard split.
Why it mattersAn open, deliberately hard multimodal eval with ground-truth responses rather than multiple choice, plus a documented finding that hard prompts increase human rater disagreement and complicate partial credit.
Musings On Building A Generative Ai Product
LinkedIn's engineering account of shipping a generative AI feature.
Why it mattersA candid build log for retrieval-grounded agent products, including how queries get routed to specialist agents and which parts of quality proved hardest to reach.
Reka Core Our Frontier Class Multimodal Language Model
Reka's flagship multimodal model, trained from scratch, lands with 128K context and native image, video and audio understanding.
Why it mattersA frontier-class multimodal model with 128K context and image, video, and audio input ships via API, on-prem, or on-device, offering a rare deployment-flexible alternative when data cannot leave your infrastructure.
Debugging AI With Adversarial Validation
Adversarial validation as a simple way to detect drift.
Why it mattersA cheap, infrastructure-free way to catch the silent killer of eval validity — your eval set no longer matching what production actually sends your model.
Diffusion Models for Video Generation
Diffusion models moved from image synthesis to video, a strictly harder problem because temporal consistency across frames demands world knowledge a single image never needed.
Why it mattersA structured survey of how diffusion models extend from image to video generation, unpacking the temporal-consistency and data-scarcity problems that define the current research frontier.
Task-Specific LLM Evals that Do & Don't Work
Evals for classification, summarization, translation, copyright regurgitation, and toxicity.
Why it mattersIt breaks down which eval approaches actually work for specific tasks like summarization, translation, and toxicity detection.
Your AI Product Needs Evals
Hamel Husain's argument that unsuccessful LLM products almost always share one root cause, the absence of a robust evaluation system, drawn from five years of building and consulting on them.
Why it mattersThis is the reference playbook for eval systems — the single highest-leverage practice separating LLM products that improve past the demo stage from ones that stall.
Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters
Qwen1.5-MoE-A2.7B is a small mixture-of-experts model that matches 7B performance with about a third of the parameters activated, part of the wave of MoE work that followed Mixtral.
Why it mattersQwen1.5-MoE-A2.7B delivers roughly 7B-class quality while only activating 2.7B parameters, meaning cheaper and faster inference for anyone self-hosting or running cost-sensitive LLM workloads.
Is Fine-Tuning Still Valuable?
Hamel Husain's answer to growing disillusionment with fine-tuning: it remains valuable in specific situations, and the case against it usually generalises from the wrong ones.
Why it mattersGives a clear decision framework for whether to invest in fine-tuning.
Finding Ai Generated Deepfake Faces In The Wild
LinkedIn and Hany Farid describe a detector for AI-generated faces that aims to generalize across generator families, not just the one it was trained on, extending earlier work that caught 99.6 percent of StyleGAN profile photos under a strict false positive budget.
Why it mattersSynthetic profile photos are cheap to produce, and this documents a detection approach that holds up across different image generators while keeping false positives low at platform scale.
What I learned from looking at 900 most popular open source AI tools
What a survey of the 900 most popular open-source AI repositories shows about how the ecosystem has changed, four years after the same exercise on the open ML landscape.
Why it mattersA structured, data-driven map of the open source AI tooling landscape — organized into infrastructure, model development, and application layers.
An index of the vibe-coding frontier. Corrections welcome.