How we ran it
We chose 6 frontier models from various model families to test the 3 different Generative UI formats, OpenUI, Google A2UI and Vercel json-render.
For all of these combinations we used the same catalog of 70 components, built entirely from OpenUI’s public component library: the open-source chat set, six chat blocks from the same library, and twelve components from the public shadcn-chat example. The exact catalog is committed in the repo.
For each format we used that SDK’s own official prompt generator and parser, and gave all three the same two worked examples, so no format was hand-tuned against the others.
Every model ran every prompt 4 times, giving 1,104 runs per format.
In total 46 different prompts were used that mirror real life requests. None of them name a component or a layout to mimic real life user queries.
Rubrics
For evaluation we used the following rubrics:
- Structural validity: a run passes when it parses, has a root, every reference resolves, nothing is orphaned or invented, required props and enums are valid, and the output is not truncated. It also needs at least as many components as the brief has requirements; that is a count floor, not a check that each requirement was addressed. Higher is better.
- Render success: whether anything reached the screen at all, measured separately from structural validity. Higher is better.