Methodology

One catalog, the same 46 interface briefs, and a shared scoring standard.

Benchmark setup

We test how reliably models generate working interfaces, without telling them which components or layouts to use.

  • 46 briefs across five complexity bands, from simple screens to dense workspaces.
  • 70 equivalent components available to every format.
  • Four generations per brief, with a 16,384-token output limit.
  • Temperature 0.7 where supported, with minimal or no reasoning.

For the framework comparison, the same briefs run across 6 models and 3 formats: OpenUI, Google A2UI, and Vercel json-render. That is 1,104 generations per format.

Each format uses the prompt generated by its own SDK. All three receive the same two worked examples, and their component catalogs are checked against one shared reference catalog.

What counts as valid

Each format is parsed and validated by its own shipped SDK, followed by the same completeness checks for all three formats.

A generation passes structural validity when it:

  • parses and produces a root;
  • uses known components and valid required or enum props;
  • resolves every reference, with nothing orphaned;
  • does not end because it hit the output limit; and
  • contains at least as many reachable components as the brief has requirements.

Render success is separate. It records whether a non-blank screen appeared at all. A screen can render and still fail structural validity.

Reproduce the results

Raw model outputs and scored verdicts are committed to the benchmark repository. The scorer runs offline, so the published outputs can be rescored without model API keys. Recreating the committed result files with no diff is the integrity check.

The repository README contains the exact Node and A2UI Python setup, plus commands for rescoring one model or the full benchmark.

Read the reproduction instructions or inspect the published raw outputs.

Important notes

Every row is scored by one build of the shipped OpenUI parser, lang-core 0.2.16, whose parser validates enum and scalar prop values. Raw outputs and per-run verdicts are committed for independent rescoring.
  • The component-count floor prevents tiny valid outputs from passing, but it does not judge visual quality or prove every requirement was understood.
  • Cost uses measured tokens and list prices. Hidden reasoning tokens can make the actual bill higher.
  • Self-hosted model costs are marked as not comparable rather than treated as free.
  • OpenUI built this benchmark. The method and outputs are published so the results can be inspected and challenged.