> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myustadia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Voice WebSocket

> Real-time spoken intake over one WebSocket: tickets, audio frames, transcripts, turns, and barge-in.

Voice mode runs the **same assistant, the same questions, the same session** as the
text intake. A client can start a session in chat, continue it by voice, and switch
back, and the conversation history follows. Everything on this page is the wire
contract; the machine-readable source of truth is
[`docs/asyncapi/voice.yaml`](https://github.com/MyUstadia/MAHARA/blob/main/docs/asyncapi/voice.yaml)
in the API repo, and this page tracks it.

## The flow at a glance

1. **Create or reuse an assistant session** (REST, same as text).
2. **Mint a single-use ticket** (REST, authenticated).
3. **Open the WebSocket with the ticket** (no Bearer header on a raw `new WebSocket()`,
   which is exactly why the ticket exists).
4. **Say hello, get ready**, then stream microphone audio and handle server frames.

```js theme={null}
// 2. mint the ticket (expires in seconds, single-use)
const r = await fetch(
  `${API}/api/v1/courses/${courseId}/assistant/${sessionId}/voice/ticket`,
  { method: "POST", headers: { Authorization: `Bearer ${apiKey}` } });
const { ticket } = await r.json();

// 3. open the socket
const ws = new WebSocket(`${WSS}/api/v1/voice?ticket=${ticket}`);
ws.binaryType = "arraybuffer";

// 4. handshake
ws.onopen = () => ws.send(JSON.stringify({
  type: "hello",
  protocol_version: 1,
  language: "fr",              // must match the session's language
  input_sample_rate: 16000,    // PCM 16-bit mono
}));
```

## Audio format

| Direction        | Format                                                                                     |
| ---------------- | ------------------------------------------------------------------------------------------ |
| Client to server | raw binary frames, PCM 16-bit little-endian, mono, **16 kHz**                              |
| Server to client | raw binary frames, PCM 16-bit little-endian, mono, **24 kHz** (`ready.output_sample_rate`) |

Send small frames continuously (20 ms frames work well). The server does the
end-of-speech detection; you never decide when an utterance is finished.

## Client to server messages

| Type                   | When                                     | Notes                                                                                                                                                                                             |
| ---------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hello`                | first message, once                      | `protocol_version`, `language`, `input_sample_rate`. Any mismatch closes 4400.                                                                                                                    |
| binary frame           | continuously while the mic is open       | see audio format above                                                                                                                                                                            |
| `playback.interrupted` | the user interrupted playback (barge-in) | `turn_no`, `played_ms` actually played before you flushed the buffer. **Implement this**: it is the only source of truth for what the user really heard, and the transcript record depends on it. |
| `text.input`           | the user typed instead of speaking       | `text`. Same turn pipeline, billed the same, `source` recorded as text.                                                                                                                           |
| `end`                  | the user hangs up                        | server closes 1000                                                                                                                                                                                |

## Server to client messages

| Type              | Meaning                                     | Client duty                                                                                                                                      |
| ----------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ready`           | handshake complete                          | carries `session_id`, `turn_no`, the current `question`, `output_sample_rate`, and `engine` (see below)                                          |
| `audio.begin`     | a spoken segment starts                     | `kind` is `"filler"` (a thinking sound, not an answer) or `"answer"`/`"nudge"`. Start your playback cursor here for `played_ms` accounting.      |
| binary frame      | speech audio                                | play it                                                                                                                                          |
| `audio.end`       | the segment finished                        |                                                                                                                                                  |
| `audio.cancel`    | **stop playback NOW and flush your buffer** | sent on barge-in. Ignoring it plays stale audio over the user. Then report `playback.interrupted`.                                               |
| `transcript.user` | what the server heard the user say          | render it in the conversation, **keyed by `turn_no`**                                                                                            |
| `turn`            | the completed turn result                   | same body as the text assistant's turn response: `message`, `question`, `accepted`, `rejected`, `coverage`, `progress`, `done`, `billed_seconds` |
| `metered`         | billing checkpoint                          | `remaining_seconds`, `percent_consumed`                                                                                                          |
| `error`           | recoverable or terminal error               | `recoverable` tells you which                                                                                                                    |
| `closing`         | the server is hanging up                    | `reason`: `idle`, `duration`, `done`, ...                                                                                                        |

## The `engine` field and live mode

`ready.engine` is either `"relay"` or `"live"` (absent means `"relay"`, older servers
do not send it). **The frame protocol is identical in both modes.** A client written
against this page works on both. Two timing rules matter, and both are safe to follow
unconditionally:

1. **Render transcripts by `turn_no`, never by arrival order.** In live mode
   `transcript.user` can arrive *after* the answer's `audio.begin`, because
   transcription streams while the model is already speaking.
2. **Never string-match `turn.message`.** In live mode it is the transcript of what
   the assistant actually said, phrased naturally, not a fixed template.

## Barge-in, the part integrators get wrong

The user is allowed to interrupt the assistant mid-sentence. The full sequence:

1. User starts talking while audio is playing.
2. Server sends `audio.cancel` for that `turn_no`.
3. You stop playback immediately, flush the buffer, and send
   `playback.interrupted` with the milliseconds actually played.
4. The conversation continues; the transcript records only what was really heard.

## Close codes

| Code | Meaning                                                   |
| ---- | --------------------------------------------------------- |
| 1000 | normal close (`end`, idle timeout, duration limit)        |
| 4001 | ticket unknown, expired, already used, or session settled |
| 4004 | session not found                                         |
| 4009 | superseded by a newer socket on the same session          |
| 4029 | voice capacity reached, try again shortly                 |
| 4400 | protocol mismatch or language not available for voice     |
| 4402 | the plan's seconds are used up                            |
| 4403 | origin not allowed                                        |
| 4500 | internal error                                            |

## Billing

Voice turns meter into the same seconds bucket as everything else. Each `turn` frame
carries `billed_seconds`, and `metered` frames surface the remaining balance so you
can warn the user before the tank runs dry.

## Sandbox

`sk_test_` sessions run the full protocol, including endpointing and frames, but
never call a vendor and never bill. Ideal for integration tests.