Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

Generating a smaller code change does not automatically make an AI editor more reliable. In a Flutter/Dart study, two small models trained to rewrite complete files outperformed versions trained to produce sequences of local search-and-replace edits across the measured outcomes.

Local edits were competitive when changes were short and confined to a small area. The authors identify longer edit sequences as opportunities for mistakes to accumulate, including replacing the wrong occurrence of repeated text. The comparison is specific to these models and this editing setup, with unequal training budgets. It suggests that output format should be evaluated alongside the size and location of the required change.

“Direct generation substantially outperforms diff-based generation on every metric we measure - compilation/static-analysis pass rate, bits-per-byte, character-level similarity to the reference, and blinded LLM-judge ratings of goal fulfillment, correctness, and code quality”

Andrej Andrejev

“it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in our dataset”

Andrej Andrejev