Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM
Source
Daniel Amyot
Author
Daniel Amyot
Date
Key takeaways · AI-distilled
The measured overhead of building with an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →: roughly 18% of the 374 human turns went to correcting the agent's own mistakes rather than to asking for anything new.
The "the agent said it works" gap closed only with oracles: make each model transformation mechanically checkable, so success is proven by a test rather than asserted in a transcript.
Every feature wave dragged a tail of consistency work behind it, updating docs, notebooks and tests to match. The test suite grew alongside, from 108 to 691 test functions.
They got numbers instead of anecdotes by mining the development record itself, 18 sessions and 10,328 tool actions over 65 hours, with a taxonomy that tags each human turn as a correction, a consistency fix, or a request the human took back.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
If you're deciding how much to trust an AI coding agent for real modeling/tooling work, this gives you actual measured overhead numbers (fix-to-feature ratio, error-correction turn rate) instead of anecdote, plus a concrete oracle-based validation strategy to catch agent mistakes before they compound.
Key quotes
“Up to version 0.7.4, fixes outnumber features 2.3:1, with ~18% of turns for correcting agent errors.”
“Feature waves dragged a measurable tail of documentation/test/notebook consistency work, and tests grew lockstep with features.”
