RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation

¹ MMLab, CUHK² Alibaba Token Hub³ The Chinese University of Hong Kong⁴ Fudan University⁵ The University of Sydney

* Work done during an internship at Alibaba Token Hub. † Corresponding authors.

GitHubcoming soon

Hugging Facecoming soon Leaderboard Human alignment

Agents develop new webpages from instructions and visual references.300 tasks · 1–5 reference webpages per task · 8 application domains

Abstract

Benchmark & Evaluation

A task combines a natural-language instruction with screenshots from one to five reference webpages. The agent must selectively interpret and compose visual properties while implementing the requested functionality; reproducing a single screenshot is not the objective.

Overview of RefCraft: (a) eight application domains and their subdomains around the project mascot, with reference-use intensity and reference-count distributions; (b) a task whose new webpage borrows a palette from reference A and a layout from reference B before the agent renders, inspects and revises it; (c) overall score against reported token use for nine frontier models, with the Pareto frontier marked. — **Overview of RefCraft.** **(a)** Eight application domains, varied reference counts, and different levels of reference use. **(b)** A task whose new webpage takes its palette from reference A and its layout from reference B, which the agent then renders, inspects, and revises. **(c)** Overall score against reported token use across the nine evaluated models. From Figure 1 of the paper.

Dataset Statistics

300tasks
186webpage types
3,491reference screenshots
12,082scored criteria

Evaluating the Complete Task

Models operate in a shared OpenCode environment. Each agent writes a self-contained HTML page and may render, inspect, and revise intermediate implementations. Browser traces and measurements expose relevant page states; programmatic checks assess deterministic criteria, while a vision-language judge evaluates semantic and perceptual requirements.

Keypoint completion
Mean normalized score across the task’s keypoints.

Core functionality
Minimum completion rate across essential functional goals.

Reference following
Mean adherence across applicable reference groups.

Score each task

S_{t} = \min (B_{t}, C_{t}, R_{t})

Average across tasks

Overall = \frac{1}{300} \sum_{t = 1}^{300} S_{t}

Each task is limited by its weakest dimension. Overall averages these task scores; R is omitted when no reference group applies.

How the benchmark is constructed

1. Visual reference pool

Licensed templates, conventional synthetic pages, and atypically styled pages provide varied layouts, palettes, typography, components, interactions, and visual character.

2. Controlled task synthesis

Each design dimension can copy, adapt, or invert a reference, follow an explicit textual requirement (specify), or remain unconstrained (ignore).

3. Verified evaluation keypoints

Requirements are decomposed into independently assessable criteria, checked through model review and revised by ten web-development experts.

Paper diagram of reference-pool curation, instruction and keypoint construction, agentic generation, and evidence-based evaluation. — **Benchmark construction and evaluation.** A diverse reference pool supports controlled user requests. Verified keypoints remain hidden from the generation agent and guide programmatic and visual evaluation of the completed webpage.

Leaderboard & Analysis

Model Comparison

9 models

B Keypoint completion · C Core functionality · R Reference following. Overall is the mean of task-level min(B, C, R), out of 100. Tokens are provider-reported means per task; accounting can differ across providers. Bold marks the best value; underline the second best. Select a column to sort, or a model to inspect its details.

Capability Breakdown

Keypoint scores averaged equally across the nine models · 0–100

By Visual Dimension

Palette92.4
Layout83.1
Components82.2
Typography76.1
Visual character58.8
Interaction43.6

By Requirement Type

Invert98.9
Specify87.7
Copy82.4
Adapt75.5

Agents transfer concrete, locally observable attributes more reliably than interactive behavior or coordinated visual character. Selectively adapting a referenced attribute is harder than reproducing, specifying, or avoiding it.

Paper §5.3.1 and Figure 3.

Agreement with Human Judgment

100 model–task artifacts · 10 web-development experts

0.747Pearson r

Task score vs. mean expert score

0.64Weighted Cohen’s κ

Keypoint label vs. median expert label

0.89Spearman ρ

Model ranking vs. Arena.AI human preferences ↗

Paper §5.3.3 · Arena snapshot September 24, 2026. The archived comparison of seven shared models recomputes the ranking correlation to ρ = 0.786; the paper reports 0.89 without listing its matched model set. Archived ranking data ↓

Interactive Examples

BibTeX

If you find RefCraft useful in your research, please consider citing our work.

Download .bib

@misc{wang_refcraft,
  title  = {RefCraft: Benchmarking Visual-Reference-Conditioned
            Agentic Generation through Web Development},
  author = {Yuchi Wang and Rongyao Fang and Weihao Yu and
            Xilin Wei and Tiancheng Gu and Dingkang Yang and
            Hang Zhang and Dayiheng Liu and Hongsheng Li
            and Shuai Bai},
  note   = {Manuscript}
}

RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development