An AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s average success rate can overstate how consistently it handles the same request. IBM reports that a GPT-4.1 agent averaged 77.4% success across repeated AppWorld runs, but passed every one of five attempts on only 53% of tasks. Its experiment measures those outcomes separately.
Building on ALTK-Evolve's existing guideline memory, the new analyzer resamples decisions from recorded runs to identify unstable choices, then generates instructions addressing them. On the same tasks, the share passed five times rose to 69%. The result supports testing repeatability alongside average performance.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
