# Talk to ask, look to choose

> When most phone interaction happens by voice, the screen stops being the app and becomes a **decision surface** the conversation summons and dismisses.

## 1. The thing that's usually diagnosed wrong

The instinct that "loading an app on screen and tapping through it is not the superior voice experience" is right. The usual diagnosis is not.

The problem is **not that UI appeared**. It's that an *app* appeared — a place you go and operate, which takes the turn away from the conversation and hands you a mode you now have to exit. Mid-sentence, that's a hard context switch with no natural way back.

Voice doesn't need less UI. It needs UI that is **subordinate to the turn**: appears mid-sentence, resolves exactly one ambiguity or gathers one approval, and collapses back into speech.

And you can't talk your way out of needing it. Speech is serial, ephemeral, ~150 wpm, no random access, no persistence. It is *superb* at the high-entropy end — "find me a fast charger under sixty cents, skip anything with a queue" is one utterance against six screens of filters — and *useless* the moment you must compare four things, pick five of twelve, verify a number, or approve something irreversible item by item.

The screen isn't a fallback for when voice fails. It's the **random-access half of a channel whose other half is serial**.

## 2. Where the three technologies sit

Decompose a turn — **hear → ground → decide → act → report** — and three things that look like competitors turn out to barely overlap.

**A2UI is a decision, not a screen.** An A2UI payload is a flat, typed, model-legible description of *a choice that needs making*. A screen is one projection of it. So is a watch, a car HUD — and so is speech.

The sharpest evidence this matters: ChatGPT's Apps SDK renders app UI as an **HTML iframe**, and the model can't see inside it — which is why the SDK had to invent `ui/update-model-context`, a side channel whose only job is telling the model what's on the screen it just rendered. That channel is a symptom. A2UI needs none of it. Apple reached the same conclusion from the other side and shipped **View Annotations** — map on-screen elements to app objects so Siri can reason about them. Correct, closed, iOS-only. A2UI is the open version, and it gets there by construction rather than by annotation.

Two consequences worth building on:

- **The data model is the antecedent store.** "The second one." "The Ionity one." Deixis is most of natural speech and none of it resolves unless the agent knows the current referent set. Rendering a surface **loads the pronouns**.
- **Voice and touch are two input drivers on one data model.** The spec already makes two-way binding *local to the renderer* — an input writes immediately, and nothing reaches the agent until an `action` fires. So an ASR write is already legal. Voice input needed no protocol change; it was always allowed.

**WebMCP is verbs at the granularity of a spoken clause.** Its value isn't driving websites — it's that a named tool with a JSON schema is *narratable*. "I clicked the third button" is not a sentence; "I moved your 3pm to Thursday" is. That its tools return plain strings looks like a limitation on a screen and is the right shape in voice. One risk that gets missed: **prompt injection is worse when nobody's looking at the page**, because the injected instruction never reaches human eyes at all.

**The host owns the loop.** Mic, endpointing, barge-in, ducking, the on-device model, the one chokepoint where consent and audit live, and one renderer across the eyes-busy surfaces — phone, watch, car, embedded — where voice is actually used. That's the job, and it has almost nothing to do with widgets.

## 3. Show or say

Speak by default. Render a surface only if at least one holds:

| Test | Trigger |
|---|---|
| **Cardinality** | more than ~3 options to compare |
| **Specificity × irreversibility** | per-item consent for something hard to undo |
| **Precision** | exact or continuous values — amounts, times, a seat map |
| **Evidence** | the user must verify something you can't summarise losslessly |
| **Persistence** | they'll act on it later, or asynchronously |

And the rule everyone violates: **never render a surface that only echoes what was just said.** A card restating "I'll move your 3pm to Thursday" is pure cost — it pulls the eyes down to read what the ears already got. If it fits in a sentence, it *is* a sentence.

## 4. Speech is the loading state

Natural turn-taking has a budget of roughly 300–500 ms. "Generate a surface, then let the user read it" does not fit inside that, which is why so much voice-plus-UI feels like a phone tree with pictures.

Two ways out, and A2UI's incremental format supports both:

- **Template-first, stream the bindings.** The skeleton is known from the catalog before the data exists, so it renders instantly and fills in while the assistant is *already talking*.
- **Run-ahead.** A routine computed the surface at 09:00 and cached it. No generation on the critical path at all.

Progressive rendering isn't polish here. A stalled screen looks slow; a stalled *conversation* sounds broken.

## 5. Why the car is the sharpest case

Everyone reaches for automotive because hands and eyes are busy. True, and not the interesting part.

The interesting part is **homologation**. Driver-distraction validation is per-HMI. If third-party UI is arbitrary code or HTML, every app and every update needs revalidation — which is why in-car app ecosystems are so thin. If UI can only be composed from a **component catalog the OEM has already validated**, the OEM validates the catalog *once* and everything an agent composes from it inherits that.

That is the only version of generative UI that survives a car safety review. "The LLM writes HTML for the centre stack" ends the meeting. "The agent may only emit components you approved, and you own the renderer" is a different conversation.

A2UI's constraint — the thing that looks limiting next to an HTML iframe — is precisely what makes it certifiable.

The same logic covers a problem cars have and phones don't: one decision must render to centre stack, cluster, HUD, or speech-only depending on drive state. That's modality-independence with a customer who has to solve it.

## 6. Choosing a speech engine

Two requirements decide it, and both are automotive-specific:

**Self-hosting.** Cars lose signal constantly and an OEM buys a licence, not a per-minute meter across a fifteen-year vehicle life. Deepgram self-hosts (Docker, Kubernetes, bare metal, SageMaker) with audio that never leaves the environment, and has Nova-3 running on the Qualcomm Hexagon NPU — the silicon path into a cockpit. ElevenLabs surfaces no equivalent.

**Named interrupt semantics.** `InjectAgentMessage` carries `behavior: default | queue | interrupt`, which is exactly the primitive the "user taps while it's talking" moment needs.

ElevenLabs still wins on voice quality and brand, which matters to a premium OEM, and has stronger native mobile SDKs. But for a car, self-hosting isn't a preference.

The stack shouldn't care either way — which is why the speech engine lives behind [an adapter](ARCHITECTURE.md#the-adapter-contract). A renderer that only works with one vendor is an integration. A renderer with a pluggable backend, where one vendor happens to be the best adapter, is a framework — and it's the more valuable position for the vendor too.

## 7. What's still missing

1. **The three `modality` gates** ([MODALITY.md](MODALITY.md)), so a renderer knows when *not* to speak.
2. **Structured returns for WebMCP**, so a tool can hand back a decision surface instead of prose.
3. **A consent and audit chokepoint for browser tools** — voice removes the human who used to be the enforcement point.
4. **A shared deixis contract** that isn't proprietary (View Annotations) or a bolt-on side channel (`ui/update-model-context`). A2UI's data model is already the right object; it needs naming as such.
5. **Mixed-modality correction as a first-class pattern** — speak, show, correct by voice, submit by tap, one data model, one undo.

Item 1 is the one this repo is about, and it's small enough to prototype without asking anyone.