Vibeleaderboard
← All Intel
Intel / video

Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust

Source
youtube.com
Author
AI Engineer
Date
From the Daily Brief

In an AI Engineer World's Fair talk, Braintrust developer advocate Jess Wang built an eval from merged fix pull requests in Microsoft's TypeScript Go repository. Claude Code had to locate the buggy code either with agentic search, meaning grep, find and reading files, or with vector search over embeddings, and the repository's own test suite scored the result. Both setups reached the same accuracy. The vector setup cost about four times as much, because retrieved chunks lacked the surrounding context and the agent kept searching to recover it. Wang's explanation is that embeddings found nearby code but missed the connective tissue between files. The talk also covers practical eval mechanics, including forcing Claude Code into vector-only search and fixing orphaned subprocess traces. This is one repository and one agent, so it is evidence rather than a rule. It does give teams a cheap test before adding an embedding index to a coding agent: build the same comparison from your own fix PRs and let your tests decide.

Read the 2026-10-06 Brief →

Context

In one Braintrust experiment, two ways of letting a coding find a bug reached the same accuracy but differed about four-fold in cost. Jess Wang, a developer advocate at Braintrust, presents it in an AI Engineer talk. Agentic search lets the agent explore with tools such as grep, find and file reading. Vector search instead retrieves code chunks by similarity, meaning numeric representations of meaning.

She built the test from fix pull requests in Microsoft's TypeScript Go repository, ran Claude Code from Anthropic with each search method, and scored results with the repository's own test suite. The description says vector search cost more because chunks lacked surrounding , so the agent kept searching, and that it found nearby code but missed the connective tissue.

This is a single experiment on one repository by a company that sells tooling, and the description gives no task count, so it does not settle which search method is better in general. The talk's caveats section was not available here.

Key takeaways · AI-distilled
  • Braintrust built the dataset from merged fix PRs in Microsoft's TypeScript Go repo, had Claude Code locate the buggy code with either grep-style agentic search or embedding search, and scored results with the repo's own test suite.
  • Vector search found nearby code but missed the connective tissue: chunks lacked surrounding context, so the agent kept searching, which is why it cost about four times as much for the same accuracy.
  • Forcing Claude Code into vector-only search for the experiment also required fixing orphaned subprocess traces.
  • Wang frames an eval as four parts: dataset, task, scorer and experiments, with observability and evals feeding each other as a flywheel.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…