A working proposal · from a running implementation
A2UI can describe a decision but cannot speak it. Deepgram can speak anything but has no idea what is on the screen. Neither gap is large — and closing them turns two products that don’t compete into one voice-first stack. This is what each side has to build.
All four are additive. None breaks an existing renderer, and the first two are the only ones that need consensus.
ComponentCommonA model reading a component tree can work out what to say. It cannot work out what happens if it says the wrong thing. Three fields, declared once per component type in the catalog:
{
"accessibility": { "label": "Pay for this charge" },
"modality": {
"requiresVisual": false, // speech can't carry it — show it or decline
"stakes": "readback", // none | confirm | readback
"spokenSensitive": false // render it, never say it aloud
}
}
They are gates, not scripts — they decide whether to speak, never what words come out. Phrasing is free at runtime; consequences are not inferable.
Precedent, not novelty. AccessibilityAttributes already occupies this slot for the same reason: non-visual metadata a renderer needs to behave correctly. A voice renderer is a screen reader that also knows the stakes. And live is already doing announce-on-change.
This is a defect in ask 1, discovered live. Asked to confirm a booking, the agent put card details into a non-sensitive component’s field, and the readback gate dutifully spoke them:
“…eighty-nine euros ninety, charged to the Visa ending 4417. Confirm?”
Nothing was violated. The gates guard components; sensitivity is a property of data. A model can defeat the gate without doing anything unreasonable. The fix is a surface-level declaration checked wherever a value is about to be spoken:
"sensitivePaths": ["/charges/*/card", "/traveller/passport"]
This is the main thing standing between the proposal and something worth upstreaming. Prompt instructions lower the frequency and fix nothing structural.
A2UI has two-way binding, and it is explicitly local to the renderer — which is exactly right. What it lacks is a convention for which row is currently chosen, so every renderer invents one and voice and screen drift apart.
A declared selectionPath on any collection component fixes it: a tap writes the index locally and the card is highlighted on the same tick, before anything reaches the agent — and a spoken pick writes the identical path.
Cheap and load-bearing. A tap that does not immediately look chosen is the most common generative-UI mistake, and it is what this implementation got wrong first.
@a2ui/voice alongside the Flutter, Lit, Angular and React renderers — plus a conformance/core/modality.yaml modelled on the accessibility suite that already exists.
The precedent is already set. @a2ui/markdown-it renders A2UI with no pixels at all, in about 33 lines. A renderer whose output device is a speaker is the same category, not a new one.
Two are documentation, two are SDK surface. None requires a new model.
behavior in the SDKThe Voice Agent wire protocol carries InjectAgentMessage.behavior with default | queue | interrupt. The SDK helper is injectAgentMessage(message) — no behavior argument. So the one primitive that makes a mid-sentence correction feel instant is stranded behind the raw socket.
Barge-in is the differentiator and it is the thing the SDK does not hand you. Local playback stop covers what a listener perceives; dropping queued server speech needs this.
The reliable browser path today is batch: record, POST to /v1/listen, get a transcript. It works, it is CORS-clean, and it verified at 0.998 confidence — but it has no interim results, and interim text is the only proof a user gets that the microphone is hearing them.
A first-class browser streaming client — mic in, partial transcripts out, no WebSocket assembly required — closes the last gap between this and a live conversation.
A silent microphone is indistinguishable from a broken app. That was the single most reported problem while building this.
agent.think is optional — you can run the Voice Agent API as ASR + TTS + turn-taking and drive the words yourself. That fact is buried in a type definition, and it is exactly what anyone with their own agent needs.
Everyone reaching for the Voice Agent API with an existing agent will otherwise end up with two LLMs producing speech in one conversation. They will disagree, and it will look like a Deepgram bug.
POST /v1/auth/grant — the endpoint that makes browser use safe — requires Member-or-higher permissions. A key scoped to use speech cannot mint the short-lived token that keeps it out of the browser.
The correct pattern currently requires a more powerful key than the wrong one. That is an adoption barrier pointing the wrong way.
€284 and “two hundred eighty-four euros” are the same value and different strings. A screen wants the first; a speaker needs the second; a model can produce both but nothing in A2UI says where the second one lives. This implementation solved it by adding a spoken field to every card — which works, and is a convention invented in one repo.
A2UI has to spec the shape. Deepgram has to be the renderer that consumes it. Neither half is useful without the other.
Feed the same A2UI payload to a screen renderer and a voice renderer and assert they reach equivalent decisions — same referents, same gates, same emitted action. That single test is what makes “modality-independent” a property rather than a claim.
The dependency is one-directional: nothing else is worth doing before the gates exist, and the gates need nobody’s permission to prototype.
| Step | Who | Unblocks |
|---|---|---|
| Gates in a catalog | anyone | Zero protocol change — catalogs already define their own properties. Running in this repo today. |
| Voice renderer, open source | Deepgram | Makes the gates real and settles their shape by hitting the walls, rather than by committee. |
behavior + streaming | Deepgram | Turns push-to-talk into conversation. |
Gates on ComponentCommon | A2UI | Lets a renderer rely on them instead of sniffing for a vendor key. |
| Sensitive paths | A2UI | Closes the hole that currently makes the readback gate unsafe. |
| Shared conformance | both | Makes modality-independence testable. |
Whoever ships the first working voice renderer effectively authors this part of A2UI. That is a better prize than a partnership announcement, and it is available now — the gates can live in a catalog today, and this repo is the proof.