The inference engine that treats GPU memory like virtual memory

What it does

vLLM is an inference and serving engine for large language models. It takes models from Hugging Face—over 200 architectures including Llama, Qwen, DeepSeek-V3, and multimodal variants—and runs them behind an OpenAI-compatible API server. The project also supports NVIDIA GPUs, AMD GPUs, x86 and ARM CPUs, Google TPUs, Intel Gaudi, Apple Silicon, and a growing list of less common accelerators.

The interesting bit

The core trick is PagedAttention, which treats the key-value cache not as a contiguous block but as a set of fixed-size pages that can be allocated dynamically, much like an operating system manages virtual memory. This lets vLLM batch requests far more aggressively than traditional inference engines without wasting GPU memory on padding and fragmentation.

Key highlights

• Born in UC Berkeley’s Sky Computing Lab and now maintained by a community of over 2,000 contributors across dozens of institutions
• Supports decoder-only LLMs, mixture-of-experts, state-space hybrids, vision-language models, embedding models, and reward models
• Quantization formats span FP8, INT8, INT4, GPTQ, AWQ, GGUF, and several vendor-specific schemes
• Distributed inference via tensor, pipeline, data, expert, and context parallelism
• Structured output generation through xgrammar or guidance, plus tool-calling and reasoning parsers

Verdict

If you are serving production LLM traffic and want the PagedAttention memory model across NVIDIA, AMD, TPU, or a half-dozen other platforms, vLLM is the community’s heavy-duty answer. If you are only looking for a lightweight client to call remote APIs, this is not it.