Real Design Benchmark – can you tell AI designs from real products?
- Source
- realdesignbenchmark.com
- Author
- ginzberg
- Date
Subframe built the Real Design Benchmark to test a narrow question: can a model design a new screen inside an existing product well enough to pass as the real thing? Each vote pairs a real product screen with an AI-built alternative made with Opus 5.5 or GPT-6.1 Sol. After thousands of votes, 57% of participants preferred the AI design, and only 59% correctly identified which screen was AI, barely better than a coin flip. The benchmark measures preference and detectability, not usability, accessibility or how well the generated screen fits a real codebase. It is still useful evidence that models can match an established visual system closely when given one. Teams deciding how much design work to hand to agents can treat it as a data point on visual quality, while keeping human review for interaction details and accessibility that a side-by-side vote cannot capture.
- To build each screen, the AI was given 4 screenshots from the existing product, its icons and fonts, and a written brief, then wrote HTML that was screenshotted the same way as the real screen.
- The team says it created test accounts with dummy data in each product, and manually checked every input to make sure nothing about the target screen leaked into the prompt.
- Beyond the 57% who preferred the AI screen, 33% preferred the real design and 10% rated them about the same, according to the author after a few thousand submissions.
- The initial results all came from Opus 5.5 at its medium effort setting, with GPT-6.1 Sol (also medium effort) added later, according to the author.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Checking sign-in…
Loading comments…