An 11B video model that interrupts, self-corrects, and stays quiet

What it does

MOSS-VL is a family of three 11B-parameter vision-language models for video understanding. The flagship MOSS-VL-Realtime consumes continuous video streams incrementally, answering questions mid-stream and deciding autonomously whether to speak or remain silent based on what it sees. MOSS-VL-Instruct handles offline long-video analysis and in-depth dialogue, while MOSS-VL-Base provides open pre-trained weights for downstream fine-tuning.

The interesting bit

Instead of the usual “watch the whole clip first, then answer” pipeline, the Realtime variant runs multimodal perception and text generation in parallel on a continuously arriving stream. It anchors every frame to absolute timestamps via special tokens and uses a custom cross-attention architecture with XRoPE to map text and video patches into a unified 3D coordinate space, letting it correct previous outputs dynamically as new frames arrive.

Key highlights

• Three open-weight 11B variants: Realtime (streaming), Instruct (offline), and Base (pre-training foundation)
• Cross-attention architecture decouples visual encoding from language reasoning to cut latency on dynamic streams
• “Proactive silence” — the model autonomously stays quiet when context is insufficient or nothing notable happens
• Quantized FP8 and NF4 checkpoints fit into a single 24 GB GPU
• First-class support in LlamaFactory and ms-swift for LoRA and full-parameter fine-tuning

Caveats

• Ships with a custom FlashAttention-3 backend in flash-attention-src/ that adds a cross_kv_boundary interface; this is derived from upstream and must be built locally rather than installed from PyPI.
• Real-time inference expects frames to arrive on the original media timeline; the README explicitly warns that playback speed must be kept at 1× to preserve stream alignment.

Verdict

Worth a look if you need an open-weight model for live video streams or long-form video QA. Skip it if you only need static image understanding — the entire architecture is optimized for temporal reasoning and streaming context.