Live voice frontend integration¶

AIHub uses GPT-Live 1 for continuous simulation voice. The simulation WebSocket requires an updated frontend. Coordinate frontend and backend deployment when migrating from the former Realtime protocol.

source is Web, Agent or TeamsApp. Simulation types are manager_feedback, peer_feedback, sales_simulation, interview_simulation and neutral_simulation. Encode query values with URLSearchParams. A new conversation ID is generated when omitted. Retain the ID returned by the server. All paths are relative to the deployed AIHub base URL, including any reverse-proxy prefix. WebSockets are documented here, not in OpenAPI/Swagger.

Authentication and startup¶

Use the same AIHub user authentication as existing chat. A browser can negotiate pairs in Sec-WebSocket-Protocol: x-api-key, <AIHub user key>, optionally client-id, <tenant client ID> and bearer, <JWT>. The server echoes the offered x-api-key protocol label, never the credential. Server clients can instead use X-API-Key and Client-ID headers. Conflicting header/protocol values and query string API keys are rejected. The OpenAI project key stays on AIHub. MongoDB tenant resolution uses the authenticated key and Client-ID; PostgreSQL uses the deployment's configured database, as for ordinary chat.

const url = new URL(`${aihubBaseUrl}/chatbot/api/v1/simulation/ws/chat`);
url.protocol = url.protocol === "https:" ? "wss:" : "ws:";
url.search = new URLSearchParams({
  user_application_id: userApplicationId,
  counterpart_user_id: counterpartUserId,
  simulation_type: "manager_feedback",
  context: JSON.stringify({voice: "brise"}),
  source: "Web",
});
const protocols = ["x-api-key", aihubUserKey];
if (clientId) protocols.push("client-id", clientId);
if (jwt) protocols.push("bearer", jwt);
const socket = new WebSocket(url, protocols);

Wait for this application event before sending microphone audio:

{
  "type": "session.started",
  "session_id": "live-example",
  "conversation_id": "f138bb08-62d5-4483-b8c6-c35bfa2a2cea",
  "audio": {"type": "audio/pcm", "rate": 24000, "channels": 1}
}

Do not send OpenAI session.start or session.update. AIHub owns instructions, model selection, history roles and tools. Startup has a 20-second timeout; sessions have a one-hour application limit. Voice cannot change during a session. The default is brise, a natural French voice with a feminine presentation. sillage is the natural French alternative with a masculine presentation. Supported voice names include the pinned OpenAI SDK's BuiltInVoice catalogue and these two documented French voices. Unsupported voices are rejected, not substituted. An explicit context.voice overrides the default: remove or replace any frontend voice: "marin" selection to use Brise. Start a new WebSocket session to apply a different voice.

Capture and playback¶

Both directions carry raw mono signed 16-bit little-endian PCM at 24,000 Hz, Base64 encoded inside JSON text frames. There is no WAV header, MP3, WebM, MediaRecorder container or binary WebSocket frame.

Use getUserMedia and an AudioWorklet. Request echo cancellation and implement resampling from the actual AudioContext.sampleRate to 24,000 Hz; requesting a sample rate does not guarantee the device uses it. Preserve resampler state between chunks. Convert float samples to signed PCM16 and encode little-endian bytes. Recommended chunks are 20–100 ms (960–4,800 bytes). Each frame must contain complete two-byte samples and at most one second of audio (48,000 bytes).

Pace input in real time, including silence while the microphone is active. Do not send a recorded file as fast as possible. Keep input and output running together; Live handles listening and speaking simultaneously. Avoid feeding speaker output back into the microphone. Bound browser buffers and monitor socket.bufferedAmount; if a connection cannot keep up, stop with a visible connection error instead of building an unbounded queue or silently losing user audio.

function sendEvent(type, fields = {}) {
  if (socket.readyState !== WebSocket.OPEN) return;
  socket.send(JSON.stringify({type, event_id: crypto.randomUUID(), ...fields}));
}
function sendPcm16(pcmBytes) {
  // pcmBytes is an even-length Uint8Array, already resampled to 24 kHz.
  let binary = "";
  for (const byte of pcmBytes) binary += String.fromCharCode(byte);
  sendEvent("session.input_audio.append", {audio: btoa(binary)});
}

On session.output_audio.delta, decode delta and queue PCM for uninterrupted playback in arrival order. Resample output if the player runs at another rate. Audio events have no timestamps or end-of-answer event. Do not wait for an old Realtime response.done or response.audio.done event to start or finish playback.

There is no Realtime VAD event, response.cancel, item truncation or buffer commit command in the browser contract. Speech can overlap intentionally. The player owns its local queue: if the UI deliberately discards buffered output, report the discarded samples. Preserve received transcript fragments even when their corresponding audio is discarded.

Browser-to-AIHub events¶

Every event requires a unique string event_id (1–128 characters). Unknown fields and event types are rejected. A JSON frame is limited to 100,000 characters.

Playback counters count 24 kHz samples, not bytes, frames or milliseconds. They start at zero for each Live session, never decrease, and their sum cannot exceed the output samples received. Send periodic updates and a final update before close. These counters help diagnose playback; transcript captions identify generated speech and are not proof of what the user heard.

AIHub-to-browser events¶

Append each speaker's transcript delta exactly, preserving whitespace and repeated words. Timing is relative to the start of the Live session. Events can arrive late and have no item ID or completed-turn marker. A pause in delivery is not a turn boundary. Use timestamps for caption placement and event_id to avoid rendering the same event twice. AIHub groups captions independently for each speaker and Live session. A gap greater than 2,000 ms between that speaker's audio intervals starts a new history message; network delivery delays and captions from the other speaker do not split it. The threshold is a grouping heuristic, so a long pause can split a single semantic response.

Each accepted fragment updates the same history document and messageApplicationId. createdAt remains the first fragment's persistence time; updatedAt advances. The message's data.voiceTurn records startMs, endMs, fragmentCount and delivery (transcribed for user speech, generated for assistant speech). History pagination and simulation analysis therefore operate on grouped turns. The WebSocket still sends the original caption events for immediate display. Duplicate event IDs are ignored for persistence within one session. Session state is bounded to 50,000 distinct caption events and 100,000 characters per turn; exceeding either limit terminates the session instead of truncating the text.

socket.onmessage = ({data}) => {
  const event = JSON.parse(data);
  switch (event.type) {
    case "session.started":
      // Store IDs, then start the application's AudioWorklet capture.
      startCapture(event.audio.rate);
      break;
    case "session.output_audio.delta":
      queueAudio(Uint8Array.from(atob(event.delta), c => c.charCodeAt(0)));
      break;
    case "session.input_transcript.delta":
    case "session.output_transcript.delta":
      appendCaption(event); // Preserve delta and timestamp interval.
      break;
    case "session.closed":
      stopCapture();
      // Drain the local playback queue before releasing the player.
      finishPlayback();
      break;
    case "error":
      showVoiceError(event.code, event.message);
      break;
  }
};

The capture, playback and rendering functions above are frontend integration points, not supplied browser APIs.

Closing, failures and reconnection¶

Stop capture, send the last playback.progress, then session.close. Keep the socket open while queued backend events drain, the provider finalizes (up to 15 seconds), and transcript persistence drains (up to 120 seconds). The application emits session.closed before closing with code 1000. Only then request simulation analysis using conversation_id; its session identifier is session_id.

invalid_client_event is recoverable: that frame was rejected and not forwarded. voice_session_failed is terminal, followed by close code 1011. Invalid identity, query or authentication input is rejected with 1008 or before the handshake. A connection that ends without session.closed has unconfirmed provider usage and may have unfinished persistence. Do not automatically replay audio.

Reconnect explicitly using the existing conversation_id when continuing the same simulation. Each connection starts a new Live session with fresh caption timing. Previously saved messages remain available for analysis, but earlier spoken turns are not automatically replayed into the new session. Preserve each session_id when requesting analysis for a specific session.

Simulation voice does not run through the Oria graph classifier. Application-owned scenario instructions and validated browser commands define its separate boundary.

Configuration and rollout¶

OPENAI_API_KEY is required. The dedicated continuous-voice selector remains llm_realtime_model; its default and OpenAI catalogue alias are now gpt-live-1 and gpt_live_1. It names the Live API model, despite the retained configuration field name. Live's delegated Responses backend uses the OpenAI mini semantic profile, preserving its configured model and reasoning effort. Voice duration and delegated backend token usage are separate costs.

The separate Realtime transcription catalogue and tenant selector have been removed. Standalone file transcription and simulation MP3 output retain their dedicated models. Their endpoints continue to operate independently.

Existing stored configuration is not updated by changing code or seeds. Before switching traffic, inspect the appropriate dry run, then apply the explicit data migration in the deployment environment:

python -m scripts.migrate_live_voice --backend mongodb
python -m scripts.migrate_live_voice --backend mongodb --apply
# For a PostgreSQL deployment, use --backend postgres instead.

This visits registered MongoDB tenants plus the provider catalogue, or PostgreSQL's configured schema and singleton config. It changes exact retired model references, removes the obsolete transcription selector, preserves unrelated/custom profiles and fails on conflicting canonical aliases. MongoDB writes compare the fields read; PostgreSQL writes use one transaction. Repeating the migration is safe. Alembic revision 20261021_28 removes the retired PostgreSQL provider columns; its downgrade restores empty nullable columns, not deleted model selections. Do not run the old application concurrently after dropping its columns. Coordinate maintenance, migration, API deployment and frontend deployment. No production configuration is changed automatically by the code migration.

TLS retains verified certificates using OPENAI_CA_BUNDLE, SSL_CERT_FILE, then certifi. The initial WebSocket handshake may make up to three attempts with exponential backoff, within one 20-second opening deadline. Permanent errors fail immediately; retry-control headers are respected. Provider reconnect/replay retries remain disabled after connection, so audio and delegated actions are never automatically replayed.

Before rollout, test microphone resampling, echo cancellation, simultaneous speech, queue limits, mute/unmute, interruptions, late captions, graceful close, network loss and simulation analysis with the actual frontend. Unit tests use simulated provider events; device/network/audio quality requires this end-to-end acceptance check.

Official references: Live migration, WebSockets, delegation.