Serving runtime that knows your TTS model has multiple stages

What it does

SGLang-Omni is a serving runtime built for audio and omni models that are themselves multi-stage pipelines—preprocessing, encoders, autoregressive engines, talkers, decoders, and vocoders. Instead of treating inference as one big black box, it owns the pipeline topology, stage lifecycle, and inter-stage transport, then exposes everything through OpenAI-compatible endpoints for chat, speech, transcription, and diarization.

The interesting bit

Each stage gets a scheduler matched to its actual workload, from SGLang-backed autoregressive scheduling to lightweight streaming vocoder loops, with tensor payloads moved across shared-memory, NCCL, NIXL, or Mooncake. That means the runtime acknowledges what most serving frameworks ignore: a vocoder and an autoregressive LLM have utterly different compute patterns and should not share a queue.

Key highlights

• Ships with an OpenAI-compatible surface covering /v1/audio/speech, /v1/audio/transcriptions, multimodal chat, batching, streaming, and voice upload
• Supports a wide model roster including Qwen3-Omni, MiniMax Music 3, Higgs Audio v3, MOSS-TTS, Fish Speech S2-Pro, Qwen3-ASR, and MOSS-Transcribe-Diarize
• Includes its own Omni Router for multi-worker front-door routing, health checks, and capability discovery
• Composes with the core SGLang engine for autoregressive execution where applicable, rather than reinventing it
• Transport layer handles shared-memory, NCCL, NIXL, and Mooncake backends for stage-to-stage tensor handoff

Caveats

• Still early: latest release is v0.1.1, so expect interfaces and internals to shift
• Several cookbook guides are explicitly flagged as experimental or research-oriented
• The README describes architecture in depth but stays silent on latency, throughput, or cost benchmarks

Verdict

Worth a look if you are building production infrastructure around open speech, TTS, or omni models and are tired of duct-taping preprocessors to vocoders. Skip it if you just need a quick Python script to call a single-model API.