Layer streaming: the 8B-model diet for 4 GB GPUs

What it does Soup is a Python CLI that abstracts the usual PyTorch/Transformers/PEFT boilerplate into a single YAML file. It handles quantization, batch sizing, and GPU detection automatically, then trains LoRA adapters locally. The layer-streaming mode (beta) keeps the frozen base model in system RAM or on disk, feeding the GPU one decoder layer at a time so an 8B parameter model can train on a 4 GB consumer card.

The interesting bit The author developed and measured everything on a 4 GB RTX 3050 laptop—then validated the same 3.32 GB peak memory and throughput on an H100, confirming the bottleneck is host-to-device transfer, not the GPU itself. That same validation run caught a silent gradient bug in bitsandbytes on 32B+ models that produced healthy-looking loss curves while computing wrong gradients.

Key highlights

• One YAML config drives SFT, DPO, ORPO, SimPO, KTO, and full fine-tuning (lora.r: 0).
• Layer streaming supports preference losses by reusing the same streamed base as a reference model, avoiding a second full copy in memory.
• soup ship bundles offline regression suites (MCQ, arithmetic, tool-calling, JSON validity, safety) and can post SHIP / DON’T SHIP verdicts to PRs.
• soup reward synth generates deterministic Python reward functions from reference data, with a mandatory calibration report that rejects weak verifiers.
• v0.73.0 fixed several broken distributed backends (DeepSpeed, SGLang, FSDP2) that had never actually executed a training run.

Caveats

• Layer streaming is explicitly beta, and the headline 119.6 tok/s figure was measured before a v0.73.0 correctness repair that cost ~4.8% throughput on larger models; it has not been re-benchmarked on the 4 GB card since.
• Python 3.13+ is blocked because PyTorch wheels can crash in native extensions before Soup even starts.
• Adapters trained with v0.72.0’s layer streaming were saved under broken .inner. keys and are effectively inert; they need to be retrained or re-saved.

Verdict Ideal for developers who want to fine-tune open models on local consumer hardware without wrestling with accelerate configs. If you are already running multi-node H100 clusters with hand-tuned DeepSpeed stages, this is probably too much abstraction.