# Architecture

## Three tiers, not two

The usual confusion is that "the speech engine is in the cloud" gets mistaken for "the renderer is in the cloud." They're separate decisions.

```
                       ┌─────────────────────────────────┐
   CLOUD               │  Agent                          │
                       │  writes the A2UI surface AND    │
                       │  speaks its own sentence        │
                       └────────────┬────────────────────┘
                                    │  createSurface
                                    │  updateComponents
                                    │  updateDataModel
   ┌────────────────────────────────▼────────────────────────────────┐
   │ HEAD UNIT (client)                                              │
   │                    ┌───────────────────┐                        │
   │                    │  ONE data model   │                        │
   │                    └────┬─────────┬────┘                        │
   │                  binds  │         │  binds                      │
   │        ┌────────────────▼──┐   ┌──▼───────────────────┐         │
   │        │ Visual renderer   │   │ Voice renderer       │         │
   │        │ Flutter / Lit /   │   │ @a2ui/voice          │─────────┼──▶ Speech
   │        │ React / DOM       │   │                      │         │    engine
   │        └────────┬──────────┘   └──────────┬───────────┘         │   cloud OR
   │                 │                         │                     │   on-box
   │            ┌────▼─────┐             ┌─────▼──────┐              │
   │            │ Screens  │             │ Speaker+mic│              │
   │            └──────────┘             └────────────┘              │
   └─────────────────────────────────────────────────────────────────┘
                    │                         │
                    └────────┬────────────────┘
                             ▼
                    action — a tap and "the second one"
                    send the identical event
```

**The renderer is client-side, both halves.** The audio half has no choice — mic and speaker are physically there, and a tap must stop speech in well under a human's perception of "instant". The deciding half belongs next to it because **the data model is renderer-side**: input writes are local, and nothing reaches the agent until an `action` fires. Two renderers sharing that model on opposite sides of a network is a distributed-state problem invented for nothing.

So the voice renderer isn't a separate service. It's **a second output driver on the same renderer state** — one model, two projections.

**Only the speech engine moves.** Cloud or on the head unit; nothing else in the picture changes. That's the whole reason it sits behind an adapter.

One honest exception: an assistant with no app on the device (ChatGPT voice) would render to audio server-side and stream it down. That works, but you lose barge-in quality and can't coordinate with a screen you don't own.

## The modules

| File | Does |
|---|---|
| [`src/data-model.js`](../src/data-model.js) | JSON Pointer store with the spec's upsert semantics. ~90 lines. |
| [`src/gates.js`](../src/gates.js) | Resolves the three `modality` gates, catalog → instance → defaults. |
| [`src/speakable.js`](../src/speakable.js) | Component tree → what's speakable, plus the **referent set** and phrase resolution. |
| [`src/voice-renderer.js`](../src/voice-renderer.js) | The renderer. Consumes A2UI messages, drives an adapter, emits `action`. |
| [`src/adapters/*.js`](../src/adapters/) | Speech backends. The seam. |

Two rules the renderer holds to:

**It does not run a language model.** Two models with different context both producing speech in one conversation will eventually contradict each other. The agent decides what to say; the renderer decides how it leaves the speaker, and handles only the mechanical announcements — a value that changed after the agent stopped, a readback before a commit, a confirmation.

**It does not invent an action type for speech.** "The second one" dispatches the identical `action` a tap would have. The agent can't tell which happened and doesn't need to.

## Referents: why rendering helps even when nobody looks

Deixis is most of natural speech, and it only resolves against a known set of things. `collectReferents` walks the surface for any component carrying an `items` binding — **structurally, never by component type name**, because catalogs belong to whoever owns the design system and a renderer that looks for `"List"` works on exactly one of them.

Each item becomes a referent carrying its index, title, gates, and the action a tap on that row would have fired. Resolution then tries ordinals (`"the second one"`), then `last`, then label matching, longest title first so "Ionity Mitte" beats a bare "Ionity".

One trap worth naming: **the cardinals `one`, `two`, `three` must not be treated as ordinals.** "The second one" ends in a pronoun, and registering `one → index 0` makes every such phrase resolve to the first item.

## The adapter contract

```js
{
  name: string,
  capabilities: { duplex: boolean, offline: boolean, localRecognition: boolean },

  speak(text, { interrupt?: boolean }): Promise<void>,   // must never reject
  cancel(): void,                                        // instant, local
  listen(onFinal, onPartial?): void,
  stopListening(): void,
}
```

`speak` must **never reject**. A speech engine that hiccups must not take the turn down with it — the agent's answer still has to reach the user, and on a screen-bearing host it already has.

`cancel` must be **local**. Whatever it also tells the server, the user perceives the local audio stopping, and that has to happen on this tick.

Two adapters ship:

| | `web-speech` | `deepgram` |
|---|---|---|
| Setup | none | API key + token endpoint |
| Barge-in | `cancel()` is instant and local | `behavior: 'interrupt'` + local stop |
| Offline | no (Chrome streams recognition audio) | yes, self-hosted |
| Voice quality | OS voices | Aura-2 |
| Tested here | yes, end to end | protocol-accurate, not run against a live key |

That table is the point of the seam. The argument this repo makes doesn't depend on a vendor, so the demo doesn't start with one — and if swapping them is a one-line change, the boundary is in the right place.

## Deepgram configuration, which is not a detail

The Voice Agent API bundles its own `think` LLM. **Do not use it here.** The A2UI agent is the brain: it decides what to say *and* emits the surface. So the adapter configures Deepgram as ASR + TTS + turn-taking only and speaks text handed to it via `InjectAgentMessage`. Leaving `think` active gives you two brains, and they will disagree.

The agent produces **two channels in one turn**: text to the speech engine, A2UI to the renderer. That's what `demo/agent.js` returns, and it's why most surfaces need no "conversion to speech" at all.

## Known limits

- The demo catalog flattens A2UI v1.0 **collection scopes** into `itemTitle` / `itemDetail` so the reference renderer stays readable. Real scopes resolve relative paths per item; only `collectReferents` would change.
- Phrase resolution is deterministic string matching, not a model. It handles ordinals, labels and `last`. It does not handle "the cheapest one" — that needs either a superlative pass over the bound data or a round trip to the agent via `callAgentFunction`.
- `confirm` and `readback` currently differ only in how much detail is restated. A real implementation would also differ in timeout and in whether an ambiguous answer re-prompts or cancels.
- No partial-transcript handling. Real barge-in wants a partial utterance to duck the agent before the final transcript lands.