Does It Render Everywhere? A Study of Cross-Environment Compatibility in MLLM-Generated Webpages

An AI-generated webpage can resemble its target screenshot and still break on another browser or device. The WebCompat study examined pages from eight AI tools across nine browser-and-device combinations. Its authors report that 68% of the studied pages had at least one compatibility problem, including layouts that shrank text excessively or cut off content.

The finding exposes a limitation of judging generated pages in one fixed viewing environment. The researchers also describe a detector that combines screenshots with the page's structural representation to flag compatibility problems. Its role is detection; the reported result does not establish automatic repair or the failure rate of AI-generated websites generally.

“Multimodal Large Language Models (MLLMs) have been increasingly adopted to automate webpage generation from visual designs (e.g., screenshots). However, existing evaluations are limited to visual fidelity assessment under a fixed browser-device configuration. Such a setting overlooks the cross-environment rendering compatibility for real-world deployments.”

Ziyun Guo, Jingyu Xiao, Yuqiang Sun, Yintong Huo

“Specifically, we construct WebCompat, a dataset of 2,032 annotated instances, comprising webpages generated by 8 representative AI tools, each rendered across 9 browser-and-device combinations.”

Ziyun Guo, Jingyu Xiao, Yuqiang Sun, Yintong Huo

“Our findings reveal that 68% of generated webpages exhibit at least one compatibility issue, underscoring the pervasive reliability concerns surrounding MLLM-generated front-end artifacts.”

Ziyun Guo, Jingyu Xiao, Yuqiang Sun, Yintong Huo

“The most prevalent symptoms are failures that disrupt the entire page layout (88.3%): pages shrink directly to fit the target screen with too small fonts, or exhibit scale mismatches that produce cut-off content.”

Ziyun Guo, Jingyu Xiao, Yuqiang Sun, Yintong Huo

“Guided by the findings, we develop XCompat, a lightweight offline compatibility issue detector that combines visual screenshots and the structural DOM tree for analysis. It achieves an F1 score of 0.903 on the WebCompat-test, outperforming the existing compatibility checking tools and LLM baselines.”

Ziyun Guo, Jingyu Xiao, Yuqiang Sun, Yintong Huo