Your model has 89 model dependencies and you cannot see them
- Source
- x.com
- Date
LLMs are no longer created w/ human data alone. They rely on other models to generate & filter data, evaluate outputs, & guide dev work. So what is a modern LLM built on? Olmo 3 → 89 model + 183 dataset dependencies; Nemotron 3 → 273 + 560 We made ModSleuth to trace this. 🧵
Modern LLM dependencies are scattered, recursive, & hard to see. So how do we even find them all? ModSleuth helps by reading papers, model & dataset cards, code configs, & upstream artifacts, then reconstructing a model's “family tree.”
Some dependency chains go 8 hops deep—a web of models & data that contributed to an LLM’s core. Turns out AI supply chains may be more tangled than we thought.
A model's lineage is broader than its training data, & every step can affect what – and how – the final model learns. Without provenance, it's harder to know where model dependencies came from, whether benchmark scores are accurate, & which upstream licenses/terms may apply.
Context
A model can depend on another model without inheriting its weights: the upstream system might generate training examples, filter documents or judge candidate answers. ModSleuth, introduced by Ai2, reconstructs these relationships by reading public papers, model documentation and code, then following dependencies through successive releases. The research paper describes an evidence-linked graph of that supply chain.
This makes it possible to investigate shared ancestry between training and systems, inconsistent documentation and potentially relevant upstream usage terms. The graph is bounded by what public sources disclose. Its value is making documented relationships inspectable, rather than treating a model's own training-data list as its complete provenance. Scope and findings.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…