A training framework that claims to make QLoRA obsolete

What it does Surogate Trainer is a C++/CUDA engine for pre-training, fine-tuning, and reinforcement learning of large language models. It supports full fine-tuning, LoRA, and QLoRA variants across single-GPU, multi-GPU, and multi-node setups. The framework targets NVIDIA GPUs from Ampere (SM80) through Blackwell (SM100+), with explicit support for models like Qwen3, Llama 3.x, GPT-OSS, and Nemotron.

The interesting bit The project makes the unusually specific claim that its native CPU offloading achieves “superior performance and VRAM usage compared to QLoRA” at native bf16 precision, “rendering QLoRA obsolete.” It also ships native FP8 and NVFP4 (E2M1) training recipes with per-tensor delayed scaling and stochastic rounding—precision levels typically reserved for inference, not training. A Python DSL with ahead-of-time auto-differentiation is offered for adding new model architectures without dropping into C++.

Key highlights

• Native FP8 (E4M3/E5M2) and NVFP4 training with CUTLASS kernels; Blackwell-specific optimizations for B200/RTX 50-series
• CPU offloading for weights, gradients, activations, and quantized states
• Built-in RL with GRPO and “predictable, deterministic environments”
• Adaptive training monitoring: auto phase detection, multi-criteria early stopping, Chinchilla token budgeting
• Stacked LoRA: train adapters on top of other adapters without offline merging
• Docker images and pre-built wheels for CUDA 12.8/12.9/13.x

Caveats

• The “surpasses all existing training frameworks” performance claim is stated without supporting benchmarks in the README; readers must visit external docs to verify
• CUDA 13.x support requires driver >=580, which is bleeding-edge; most users will need the 12.8/12.9 paths
• Model support is explicitly listed but narrow: no Mistral, Gemma, or non-NVIDIA architectures mentioned

Verdict Worth evaluating if you’re training on NVIDIA hardware and need to push VRAM boundaries with sub-16-bit precision, especially on Blackwell. Skip if you need broad model coverage beyond the supported list or run on non-NVIDIA accelerators.