Generative UI Reliability benchmark

We ran the same 46 prompts through the 3 different Generative UI formats, OpenUI, Google A2UI and Vercel json-render on 6 different models across different model families.
On completion rate, OpenUI (92.9%) and A2UI (92.4%) finished in a statistical dead heat, with json-render more than 11 points behind (81.3%), while OpenUI consumed less than half the tokens per screen.
A2UI performed the worst when it came to blank screens, with at least 53 out of 1,104 runs resulting in a fully blank screen while OpenUI only failed to render twice vs json-render's 9.

Methodology

We chose 6 frontier models from various model families to test the 3 different Generative UI formats, OpenUI, Google A2UI and Vercel json-render. For all of these combinations we used the same catalog of 70 components, built entirely from OpenUI's public component library: the open-source chat set, six chat blocks from the same library, and twelve components from the public shadcn-chat example. The exact catalog is committed in the repo. For each format we used the official prompt generator and parser from the SDK, under one uniform condition for every model: OpenUI's prompt carries its component groups, two rules and two worked examples through lang-core's official options; json-render runs catalog.prompt() with its three custom rules; A2UI runs its generator's prompt as-is. Every model ran every prompt 4 times, giving 1,104 runs per format. In total 46 different prompts were used that mirror real life requests. None of them name a component or a layout to mimic real life user queries.

For evaluation we used the following rubrics:

• Completeness: Every prompt is rendered, every reference should be resolved without silently dropping anything. Higher is better
• Empty responses: Whether the user saw anything at all. If the response completely failed to render then we count that separately from completeness. Naturally the lower the score in this rubric, the better.

Results

No format wins on every model

• GPT-5.6 Sol is nearly flawless with OpenUI at 99.5%: one dangling reference in 184 runs.
• Claude Opus 4.8 is too close to call: 97.3% A2UI against 96.7% OpenUI, inside our noise band, with json-render trailing at 87.0%. Opus's OpenUI misses are five broken references (four orphaned, one dangling) and one wrong signature, zero enum errors.
• Kimi K3 leans the other way but is also within noise: 93.5% OpenUI against 92.4% A2UI, json-render at 85.3%.
• Gemini 3.6 Flash, a small lightweight model, stays close at the top (94.6 OpenUI vs 90.2 A2UI) but json-render drops to 71.7%.
• Qwen3.8 2.4T is an exact tie: OpenUI and A2UI both land on 89.7%, with json-render at 73.9%.
• Meta's Muse Spark is A2UI's one clear win: 91.8% against OpenUI's 83.7%, almost entirely on enum discipline. Muse keeps reaching for variant names that feel natural but don't exist in the catalog ("primary", "success"), and a strict DSL rejects what a JSON prop bag absorbs.

json-render's losses have one dominant cause: of 1,399 scoring errors across the six models, 1,311 are reference-graph failures, dangling and orphaned element ids. Its flat patch-registry format makes every parent-child link a string id the model must keep consistent across the whole stream, and models lose the thread as screens grow.

We suspect heavy JSON post-training explains part of the remaining spread, making JSON scaffolding strong on some models while a compact line-based DSL wins on the rest. The key takeaway is if you use a fixed model, benchmark that model. If you support several, the average matters more and it becomes harder to achieve the highest quality across various different model families.

Screens get harder as they grow

Every format degrades as generations become complicated to involve more components (which is expected).
json-render falls fastest and furthest as requirement counts climb.
OpenUI & A2UI closely tail each other across every band.

Note: prompt content moves these numbers about as much as format choice does. One rule telling the model to "attach every component it defines" was worth 13 points to OpenUI on Kimi in earlier runs, and a single wrong worked example in our own prompt cost two models double digits until we caught it. Every model+format combination needs prompt care to get the most out of it.

When it fails, what does the user see?

The completion rubric treats a missing component and a fully blank screen the same but in real life your users wouldn't. We added a rescoring script to analyze how many of the generations actually resulted in a fully blank screen from the user's perspective.

OpenUI failed to render only 2 times out of 1,104 runs, and both were empty API responses to the same disaster-response prompt, one from Muse and one from Qwen. Both models answer that prompt at full length in the JSON formats, and both answered it in OpenUI on other repeats of the same run. This looks like an intermittent provider-side safety filter that tolerates structured JSON output but sometimes suppresses freeform responses on sensitive topics; we score the empties as blanks anyway because that is what a user would see. A2UI's default renderer, which drops the whole updateComponents message if it has any invalid component structure, blanked out 56 times. Counting conservatively - rendering each component individually through the same official validator, with the all-or-nothing rule removed - 53 are still completely blank. json-render failed 9 times.

The main difference in how these formats are interpreted: each OpenUI component is one line, so a parse error only costs that line while a JSON document has no smaller unit to fall back to. This does lead to OpenUI trading blank screens for partial ones which can be harder to notice which is why we track the completion rubric separately. Rescoring scripts for both these rubrics are in the repo.

Tokens

Given input tokens & tool definitions are developer defined in all three formats, the token overhead comes from 2 categories:

• System Prompt that goes in every request: 4,828 tokens for OpenUI (including its component groups and two worked examples), 6,497 for json-render, 11,080 for A2UI for the same component library generated by the respective SDKs.
• Output Tokens become a factor of format verbosity: 1,284 mean tokens per screen for OpenUI against 3,558 for json-render and 2,740 for A2UI, a 2.1 to 2.8x gap. The same header is one line in OpenUI, a patch operation in json-render, a wrapped message in A2UI.

The same header, in each format's wire shape

# openui-lang: one line
header = Header("Executive Revenue Dashboard", "Q4 Performance Overview")

# json-render: one patch of many
{"op":"add","path":"/elements/header_main","value":{"type":"Header","props":{...}}}

# A2UI: one wrapped message
{"version":"v0.9","updateComponents":{"surfaceId":"main","components":[{...}]}}

Token optimizations are not just about output token size but a leaner format streams much faster and reduces the context size on follow up turns leading to a better UX and more turns per 100k context window.

Cost

Generating the identical 46 screens costs 1.7 to 2.9x less in OpenUI on the five models with public list prices. One pass costs $2.04 in OpenUI on Claude Opus 4.8. The same screens in A2UI cost $5.83.

Lessons from running Generative UI in production

We have been running OpenUI and its predecessor (Thesys C1) in production for a year now. Here is some of our learnings from the field:

On the older JSON format we tried few-shot examples, schema reordering, and different prompt styles. They changed which errors we saw, but not the overall failure rate. Giving the model more reasoning time helped, but added too much latency for real-time UI. That pushed us toward repairing the output instead of continuing to tune the prompt.

Overall error rates were 15% on the older json-based format which have now dropped to 4-5% after migrating to OpenUI. The error distribution from a 15-day production parser log (1,285 failures):

Most of these are mechanical. Orphans can be garbage-collected, near-miss enums snapped to valid values, typo'd references matched to the nearest defined name. None of that needs an LLM for error correction, and because every OpenUI component is one named line, a repair touches only the broken lines by reusing the same property that powers incremental editing.

The number we actually run on

Failures happen 4 to 5% of the time. The repair pipeline catches 96% of those. Multiply the two and roughly 2 screens in 1,000 (0.2%) reach an OpenUI Cloud user broken and we are actively working towards reducing that number down to 0.

Our takeaway is that raw completion rate isn't the whole story; how a format fails matters as much. OpenUI's line-level structure kept failures small enough to contain and repair, which is why we ended up building the production system around repair instead of trying to eliminate generation errors entirely. Whatever format you pick, at enough volume you'll want that repair layer too.

Footnotes:

• We built OpenUI Lang, so read this as a first-party benchmark with everything disclosed, including that the 70-component catalog is derived from our own public component library.
• A near-perfect cell means near-zero failures at this sample size, not a model that never fails; Sol's 99.5% is one failure in 184 runs.
• An extended Terra run, also OpenAI, put OpenUI first at 97.8 vs 79.9 and 79.9, but stays out of the average to keep one seat per company.
• The averages are the mean of six per-model completion rates where each model gets equal weight. Every model ran 4 generations per prompt; Gemini's json-render and A2UI legs were generated with 10 before we settled on 4 for everyone, and the scored set is the first 4 (a fixed rule, not outcome-selected).
• One condition for all models and formats. OpenUI's prompt includes component groups, two rules and two worked examples via lang-core's official generatePrompt options; json-render and A2UI ran their generators' prompts with their documented options and no worked examples. Giving the competitors equivalent worked examples through their official free-text options is planned as a follow-up, and we expect it to raise their scores.
• Scoring is each SDK's own shipped code plus one shared completeness layer with a coverage floor; the layer credits json-render's native children slot wherever a component's single ref prop allows it, and consumes A2UI validation errors its itemized checks would miss.
• The full judgment-call list, all code, prompts, and every raw output: benchmark repo.
• The method builds on Mobile Reality's MDMA benchmark.