MyUstadia API

Voice mode runs the same assistant, the same questions, the same session as the text intake. A client can start a session in chat, continue it by voice, and switch back, and the conversation history follows. Everything on this page is the wire contract; the machine-readable source of truth is docs/asyncapi/voice.yaml in the API repo, and this page tracks it.

The flow at a glance

Create or reuse an assistant session (REST, same as text).
Mint a single-use ticket (REST, authenticated).
Open the WebSocket with the ticket (no Bearer header on a raw new WebSocket(), which is exactly why the ticket exists).
Say hello, get ready, then stream microphone audio and handle server frames.

// 2. mint the ticket (expires in seconds, single-use)
const r = await fetch(
  `${API}/api/v1/courses/${courseId}/assistant/${sessionId}/voice/ticket`,
  { method: "POST", headers: { Authorization: `Bearer ${apiKey}` } });
const { ticket } = await r.json();

// 3. open the socket
const ws = new WebSocket(`${WSS}/api/v1/voice?ticket=${ticket}`);
ws.binaryType = "arraybuffer";

// 4. handshake
ws.onopen = () => ws.send(JSON.stringify({
  type: "hello",
  protocol_version: 1,
  language: "fr",              // must match the session's language
  input_sample_rate: 16000,    // PCM 16-bit mono
}));

Audio format

Direction	Format
Client to server	raw binary frames, PCM 16-bit little-endian, mono, 16 kHz
Server to client	raw binary frames, PCM 16-bit little-endian, mono, 24 kHz (`ready.output_sample_rate`)

Send small frames continuously (20 ms frames work well). The server does the end-of-speech detection; you never decide when an utterance is finished.

Client to server messages

Type	When	Notes
`hello`	first message, once	`protocol_version`, `language`, `input_sample_rate`. Any mismatch closes 4400.
binary frame	continuously while the mic is open	see audio format above
`playback.interrupted`	the user interrupted playback (barge-in)	`turn_no`, `played_ms` actually played before you flushed the buffer. Implement this: it is the only source of truth for what the user really heard, and the transcript record depends on it.
`text.input`	the user typed instead of speaking	`text`. Same turn pipeline, billed the same, `source` recorded as text.
`end`	the user hangs up	server closes 1000

Server to client messages

Type	Meaning	Client duty
`ready`	handshake complete	carries `session_id`, `turn_no`, the current `question`, `output_sample_rate`, and `engine` (see below)
`audio.begin`	a spoken segment starts	`kind` is `"filler"` (a thinking sound, not an answer) or `"answer"`/`"nudge"`. Start your playback cursor here for `played_ms` accounting.
binary frame	speech audio	play it
`audio.end`	the segment finished
`audio.cancel`	stop playback NOW and flush your buffer	sent on barge-in. Ignoring it plays stale audio over the user. Then report `playback.interrupted`.
`transcript.user`	what the server heard the user say	render it in the conversation, keyed by `turn_no`
`turn`	the completed turn result	same body as the text assistant’s turn response: `message`, `question`, `accepted`, `rejected`, `coverage`, `progress`, `done`, `billed_seconds`
`metered`	billing checkpoint	`remaining_seconds`, `percent_consumed`
`error`	recoverable or terminal error	`recoverable` tells you which
`closing`	the server is hanging up	`reason`: `idle`, `duration`, `done`, …

The `engine` field and live mode

ready.engine is either "relay" or "live" (absent means "relay", older servers do not send it). The frame protocol is identical in both modes. A client written against this page works on both. Two timing rules matter, and both are safe to follow unconditionally:

Render transcripts by turn_no, never by arrival order. In live mode transcript.user can arrive after the answer’s audio.begin, because transcription streams while the model is already speaking.
Never string-match turn.message. In live mode it is the transcript of what the assistant actually said, phrased naturally, not a fixed template.

Barge-in, the part integrators get wrong

The user is allowed to interrupt the assistant mid-sentence. The full sequence:

User starts talking while audio is playing.
Server sends audio.cancel for that turn_no.
You stop playback immediately, flush the buffer, and send playback.interrupted with the milliseconds actually played.
The conversation continues; the transcript records only what was really heard.

Close codes

Code	Meaning
1000	normal close (`end`, idle timeout, duration limit)
4001	ticket unknown, expired, already used, or session settled
4004	session not found
4009	superseded by a newer socket on the same session
4029	voice capacity reached, try again shortly
4400	protocol mismatch or language not available for voice
4402	the plan’s seconds are used up
4403	origin not allowed
4500	internal error

Billing

Voice turns meter into the same seconds bucket as everything else. Each turn frame carries billed_seconds, and metered frames surface the remaining balance so you can warn the user before the tank runs dry.

Sandbox

sk_test_ sessions run the full protocol, including endpointing and frames, but never call a vendor and never bill. Ideal for integration tests.

Get Started

Concepts

API Reference

Endpoints

Voice WebSocket

The flow at a glance

Audio format

Client to server messages

Server to client messages

The `engine` field and live mode

Barge-in, the part integrators get wrong

Close codes

Billing

Sandbox

​The flow at a glance

​Audio format

​Client to server messages

​Server to client messages

​The engine field and live mode

​Barge-in, the part integrators get wrong

​Close codes

​Billing

​Sandbox

The flow at a glance

Audio format

Client to server messages

Server to client messages

The `engine` field and live mode

Barge-in, the part integrators get wrong

Close codes

Billing

Sandbox