An inference engine so specialized it only runs on one GPU

What it does NInfer is a from-scratch C++/CUDA inference engine built for exactly one target: NVIDIA’s GeForce RTX 5090. It serves a closed registry of Qwen checkpoints—handling text, image, and video prompts—through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs. Rather than acting as a general model runtime, it behaves like a purpose-built executable for a specific GPU and a specific set of weights.

The interesting bit The project treats portability as a bug. It hard-codes support for exactly five Qwen artifacts, uses custom .ninfer weight containers instead of Safetensors or GGUF, and the build system actively rejects CUDA architectures other than sm_120a. That narrowness lets it deploy aggressive quantization schemes—NVFP4, W4A4 Tensor Core MMA, mixed FP8 allocations—without portable fallback code slowing things down.

Key highlights

• Benchmarked aggregate decode throughput hits 1,313.8 tok/s for Qwen3.6-35B-A3B and 1,146.9 tok/s for Qwen3.6-27B NVFP4 at batch-8 concurrency on an RTX 5090.
• Ships with both a local CLI (ninfer) and an HTTP server (ninfer-serve) speaking OpenAI- and Anthropic-compatible APIs.
• Supports multimodal input (image and video) via FFmpeg-backed vision pipelines, though vision weights must be loaded at startup and cannot be toggled per-request.
• Evaluation scores are published for AIME 2025/2026 and GPQA-Diamond, with Qwen3.8-27B NVFP4 scoring 96.67% on AIME and 90.40% on GPQA-Diamond.
• Uses Multi-Token Prediction (MTP3) speculative decoding with measured acceptance rates up to roughly 90% on structured-output tasks.

Caveats

• Hardware support is literally one GPU. The build rejects any CUDA architecture other than sm_120a, and there are no packaged binaries—you build from source.
• Model support is a closed list of five Qwen artifacts; if your checkpoint is not in the registry, it will not run.
• Qwen3.8-27B NVFP4 shows noticeably lower MTP acceptance (~46–49%) compared to the other profiles (~67–71%), and its groupwise-int variant is supported but not yet benchmarked.
• Capabilities like vision and speculative decoding are fixed at process startup; you cannot enable them on an already-running server.

Verdict Worth a look if you own an RTX 5090 and want to serve Qwen3.6 or Qwen3.8 checkpoints at the highest possible single-GPU throughput. For anyone lacking that exact GPU—or wanting to run arbitrary models—this is strictly a spectator sport.