Reinforcement Learning for Design Taste — Jasper Lu

Jasper Lu's experiment suggests that reinforcement learning can teach a model design taste, but the taste it learns is the judge's. Reinforcement learning improves a model by rewarding good attempts, which works well for math and code because answers can be checked automatically. Design has no such check, so the usual workaround is to have another AI model score the output. Lu trained Qwen3.8-27B on web design tasks from environments Xiaomi open-sourced alongside MiMo-V2.6, ran the training twice with identical settings, and changed only the judge: GPT-6 Sol in one run, Claude Sonnet 5.5 in the other.

Each trained model's best checkpoint beat the untrained model under both judges, but the two drifted in different directions. Lu describes the Sol-trained model as converging on wackier pages with playful fonts, emojis and background images, many with clipped or overflowing text. The Sonnet-trained model converged on what Lu calls classic Claude design traits, such as serif display fonts, muted palettes and small labels above each heading, with little variation between briefs. When the two final models were compared head to head, both judges preferred the Sonnet-trained one, by a net 44% when Sol judged and 60% when Sonnet judged. A rubric grader that Xiaomi shipped with the environments told the same story: overall scores of 64.5 for the base model, 66.8 for Sol-trained and 70.6 for Sonnet-trained, with the Sol-trained model's layout integrity falling from 73.1 to 64.7.

The run also showed how a judge's quirks become the model's habits. Lu measured that early in training both judges favored longer pages within each group, which he calls the most likely reason the models' pages kept growing until, by step 10, more than half of attempts were cut off by the output limit and delivered nothing. Lu says he has no firm explanation for why Sol trained a worse model, and offers two guesses: judging eight full-page screenshots at once may be too hard for it, and the training reward asks the judge to pick pages that are clearly better or worse than the rest of a group, which favors pages that simply look different and can reinforce eye-catching tics. He also says he is not a designer and finds the Sonnet-trained pages safe and bland.

Lu's conclusion is that the reward should split two questions the current judge blends together. An absolute rubric would check whether a page is well built, and only pages that pass would go on to a relative comparison of which looks better, possibly with a penalty for common AI design tells. He argues that taste itself may not be delegable to one model at all, citing a study by Contra that found human designers agree on pairwise choices as little as 60% of the time. These results come from one 24-step run per judge on a 32-brief evaluation set, so they show how judge choice shapes training rather than settling which judge is best. The full write-up, with the judge prompts and per-step charts, is on Lu's blog.