Vibeleaderboard
← All Intel
Intel / article

Agent Plasticity: Measuring Self-Improvement Through Experience

Source
arxiv.org
Author
Harman Singh, Anton Bakhtin, Rulin Shao, Gabriel Synnaeve, Ilia Kulikov, Rob Fergus, Sanjeev Arora, Kurt Keutzer, Jason Weston, Anuj Mahajan, Anirudh Goyal
Date

Context

Existing evaluations largely measure what an agent can do at one moment, not how well it learns. A paper proposes agent plasticity: the efficiency with which an agent converts experience into gains on future held-out tasks (tasks kept out of what it learned from), accounting for the cost of learning. In its , agents distill past experience into reusable artifacts that future instances inherit.

Across multiple environments, the authors report that frontier models improved very differently despite comparable chances to learn: some made large, lasting gains while others stayed near or below their starting level, and gains on -style tasks often transferred only partly to out-of-distribution ones. The best final performer was not necessarily the one that improved most efficiently.

Tracing failures, they found low-plasticity agents often failed to reuse relevant artifacts, while more plastic ones could still fail despite reusing them, pointing to artifact quality, generalization or . The abstract names no models or environments, so it supports the pattern, not a ranking.

Key takeaways · AI-distilled
  • In the authors' setup, agents distill past experience into reusable artifacts inherited by future instances, and each checkpoint measures training and held-out performance while accounting for learning cost.
  • Frontier models showed sharply different improvement trajectories despite comparable chances to learn: some made substantial, persistent gains while others stayed near or below their starting performance.
  • The authors report that gains within the training regime often transferred only partially to out-of-distribution conditions.
  • Endpoint capability and learning efficiency diverged: the agent that performed best at the end was not necessarily the one that improved most efficiently.
  • Tracing failures, low-plasticity agents often failed to reuse relevant artifacts, while more plastic agents sometimes failed despite reuse, pointing to limits in artifact quality, generalization, or application.
Terms in this piece · Glossary
  • Agent — A system in which a model repeatedly chooses operations, a harness carries them out, and the results return to the model to inform its next choice.
  • Setup — The combination of a model, a harness, its connections and the path the task's data takes, considered together for a task.
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
  • Application — Software arranged so that a person can use it; in this course, the software around a model that accepts your request, prepares the context and shows the reply.
Recommended reads
Comments

Checking sign-in…

Loading comments…