Dillon DuPont develops software and models for agents that operate computers. His work includes GPT-4V-Act, coauthorship of Windows Agent Arena and a small form-decision model at Cua.[1][2][3]
Start with How Cua Replaced Full LLM Calls with Fast Decision Models →
X Career
DuPont’s public GitHub profile links his work to Cua and MIT. MIT’s Isola laboratory lists him among its undergraduate alumni. His projects connect interface perception, action selection and evaluation, with explicit boundaries between what a model decides and what the surrounding software controls.[4][5]
Contributions
Give screen controls stable names
GPT-4V-Act connects a vision model to a browser through screenshots and mouse and keyboard actions. Its DOM-based labeler assigns numbers to interactive elements, so a model can refer to a control by number rather than invent its screen position. The repository distinguishes its implemented features from unfinished ones, including partial typing support and absent scrolling. A later update directs readers to Windows Agent Arena.[2]
Evaluate agents in a desktop environment
DuPont coauthored Windows Agent Arena, a benchmark for agents performing tasks in Windows applications. The paper combines a task environment, automated evaluation and parallel execution on Azure. Its authors investigate multimodal agents in operating-system workflows rather than restrict evaluation to a text-only interface. DuPont is listed with Microsoft in the paper; this establishes the research affiliation at publication.[3]
Separate choosing from executing
In a Cua article coauthored with Francesco Bonacci, DuPont describes CUA-S1-FORMS, an independent specialist he trained from ideas and code in jevlike. It scores actions for structured form elements supplied with extracted values. It does not read screenshots or discover values itself. The article contrasts this narrow experiment with jev-use, where application code builds allowed actions, a decision model chooses an ID, and the application executes and verifies the result. Neither a high score nor a selected action establishes successful completion.[1]
Tools, writing & talks
Writing, research & talks
Start here
Explains bounded action selection and DuPont’s independently trained forms specialist.
Begin with the form example and the distinction between choosing an action and checking its result. The article is readable without running code; the linked recipe requires Cua Driver and, for live Jev decisions, a TypeSafe key.
Coauthor[1]Original source- Windows Agent Arenaarxiv.org
A benchmark and infrastructure for evaluating multimodal agents in Windows applications.
Coauthor[3]
Tools
- GPT-4V-Act
github.comAn experimental Chromium assistant using numbered screen elements and vision-model decisions.
Repository owner and developer[2]