Intel / video

An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases

Source: youtube.com
Author: AI Engineer
Date: Published Sep 26, 2026

Context

ARIA, the research inside Weights & Biases, connects experiment execution with analysis: it launches and monitors training jobs and summarizes results and patterns across runs. That gives researchers a way to ask questions about experiments and start further work within the same system. The presentation description also explains how its developers evaluate the agent itself.

They log its interactions, use automated judges on live traffic to flag signals such as user frustration, and run a nightly suite of repeatable tasks written like unit tests to inform release decisions. The speaker also advises keeping humans reviewing traces. The useful pattern is to evaluate whether the agent performs research tasks reliably, alongside evaluating the models those experiments produce.

Key takeaways · AI-distilled

Weights & Biases' ARIA agent launches and monitors training jobs, summarizes the best runs, finds patterns across 200+ experiments and builds reports; Tim Sweeney demoed it running a live batch on Karpathy's autoresearch project on real GPUs.
W&B logs 100% of ARIA's traces to its Weave tool and runs judges on live traffic to flag behavioral signals such as user frustration.
The team writes agent tasks as YAML "unit tests" and uses a nightly eval suite to make go/no-go decisions on new candidates; the talk cites a 73% versus 72% comparison for the latest candidate.
Sweeney's advice for productionizing agents: invest in agent-focused observability to catch behavioral bugs, treat evals as your new CI, keep humans reviewing traces, and add and tools before over-engineering the .

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Read the source www.youtube.com