A vLLM engine small enough to read in one sitting

What it does

miniVLLM is a from-scratch reconstruction of the vLLM inference engine that runs a batched LLM generation pipeline end-to-end. It creates a small Qwen3 model with random weights, feeds it 60 batched chat prompts, and handles the full prefill-and-decode cycle using custom paged KV cache management and attention kernels. The repo also ships paired benchmark scripts that pit standard PyTorch attention against memory-efficient Flash Attention (during prefilling) and against custom Triton paged-attention kernels (during decoding).

The interesting bit

The project is explicitly pedagogical: the README points to a step-by-step implementation guide covering layers, scheduling, CUDA graphs, and the KV cache block manager. Instead of treating vLLM as a black-box dependency, the included benchmarks let you compare naive O(N²) PyTorch attention against memory-efficient Flash Attention and custom Triton kernels in the same codebase to see exactly where the memory and latency gaps open up.

Key highlights

• Self-contained Triton kernels for both Flash Attention (prefill) and paged attention (decode), benchmarked side-by-side with naive PyTorch versions.
• Full engine architecture exposed: model layers, sequence scheduler, block manager for GPU KV cache, and a generation runner.
• Includes HowToApproachvLLM.md, a guided walkthrough for building the stack from scratch.
• Runs fully offline with a randomly initialized small model for the demo.
• Multi-GPU support is configurable by changing a world_size parameter.

Caveats

• Requires Python 3.11 exactly and a CUDA GPU; it will not run on CPU or newer Python versions.
• The main demo uses random weight initialization, so output text is gibberish; this is a framework for studying inference mechanics, not for serving real models out of the box.

Verdict

Grab this if you are trying to understand why vLLM’s KV cache paging and flash attention matter under the hood. Skip it if you just need a production serving endpoint.