Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM
- Source
- arxiv.org
- Author
- Daniel Amyot
- Date
Context
An AI-assisted development record shows how much work can remain after an starts producing features. This experience report follows PM4Py-UCM, a tool that turns event logs into process models, built with Anthropic's Claude Code. Across 18 sessions covering 65 hours, about 18% of human turns corrected agent errors, and fixes outnumbered features 2.3 to one through version 0.7.4. New features also brought follow-up work to keep tests, documentation, and notebooks consistent. This single case makes correction and consistency work visible in the development process; it does not establish a general productivity rate for AI-assisted programming.
- The measured overhead of building with an agent: roughly 18% of the 374 human turns went to correcting the agent's own mistakes rather than to asking for anything new.
- The "the agent said it works" gap closed only with oracles: make each model transformation mechanically checkable, so success is proven by a test rather than asserted in a transcript.
- Every feature wave dragged a tail of consistency work behind it, updating docs, notebooks and tests to match. The test suite grew alongside, from 108 to 691 test functions.
- They got numbers instead of anecdotes by mining the development record itself, 18 sessions and 10,328 tool actions over 65 hours, with a taxonomy that tags each human turn as a correction, a consistency fix, or a request the human took back.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
“Up to version 0.7.4, fixes outnumber features 2.3:1, with ~18% of turns for correcting agent errors.”
“Feature waves dragged a measurable tail of documentation/test/notebook consistency work, and tests grew lockstep with features.”
Checking sign-in…
Loading comments…