We are Changing our Developer Productivity Experiment Design

Random assignment cannot correct for people or tasks that never enter a study. METR's developer-productivity update reports that some developers declined participation, or withheld tasks, because they did not want to risk being assigned to work without AI. The experiment therefore missed some of the work where participants expected the greatest benefit.

That makes its later results difficult to compare cleanly with the earlier study. Lower participant pay also affected selection, and running several agents at once complicated measurements of time spent. METR treats the data as weak evidence about the size of the productivity change and describes alternative study designs. Enthusiasm for a tool can itself change who remains available to measure its effects.