Skip to content
OtterVoiceOtterVoice

Architecture

OtterVoice is a voice-session orchestration SDK, not a model gateway. Core owns state, turns, barge-in, and usage; platform audio and models are injected.

Application UI
   ↕ events / commands
@ottervoice/core   VoiceSession · state machine · VAD · barge-in · usage
   ↕ RuntimeAdapter          ↕ Gateway client providers (user content only)
runtime-web | rn | node      App auth/policy gateway (server)
                              ↕ Provider API
                              Native Audio LLM or server ASR → LLM → TTS

Boundaries

Section titled “Boundaries”
Layer Owns Does not own
UI Events, captions, start / finish, local VAD/playback preferences Provider model, system prompt, voice, generation policy, long-lived keys
Core State machine, turns, interruption, usage, errors DOM / Expo / Node APIs
Runtime Mic, playback, optional network/storage Prompts, model choice
Gateway client provider User text/audio, history, and stream transport only Model, prompt, voice, temperature, or token ceilings
Application gateway User/session auth, policy injection, body reconstruction, quotas, audit Trusting client business policy
Direct provider Vendor protocol inside trusted Node/server runtimes Acting as an authorization boundary for untrusted clients

Fields such as audioLlmSystemPrompt and LLMGenerateInput.system remain valid when the whole session runs in a trusted Node/server runtime. Browser/app standard mode should omit them and use the createOpenRouterGateway* client factories; the server injects model, prompt, voice, and generation ceilings via createOpenRouterGateway().

Clients may still own VAD, captions, playback, and interruption UX. Backend selection, rolling ASR, eager audio-turn starts, and automatic retries can change spend; if exposed to clients, enforce per-user/session/profile budgets and rate limits independently at the gateway.

One contract, two backends

Section titled “One contract, two backends”
  • Native backend: completed turn audio enters a native Audio LLM and returns assistant text plus audio.
  • Composite backend: the same AudioLLMProvider runs ASR → LLM → TTS on a trusted server and streams input transcript, assistant text, and audio.

Clients always configure providers.audioLlm. A composite provider normally sets transcribesInput: true, avoiding a separate caption request. Add optional providers.asr when a native provider does not return the input transcript. audioLlmStartTiming either runs caption ASR and reply generation in parallel or waits for asr_final.

Create the session once at the composition root. Swap runtime or providers without rewriting UI.

Local turn detection

Section titled “Local turn detection”

TurnDetector uses microphone volume and deterministic timing only. volume detects both speech start and trailing silence locally; hybrid additionally lets ASR partial text confirm quiet speech, while the same local silence timer closes the turn. manual leaves the boundary to push-to-talk UI.

The runtime may send low-latency AudioChunk.delivery: 'stream' fragments and one immutable delivery: 'turn' snapshot after recorder flush. Core uses the snapshot for batch ASR, Audio LLM input, and user_audio_final, preventing stream resets or asynchronous blob reads from dropping the head, middle, or tail of a displayed turn.