ExLlamaV3 quantizes 70B models on a single RTX 4090

What it does

ExLlamaV3 is an inference engine and quantization toolkit for running large language models locally on modern consumer NVIDIA GPUs. It introduces the EXL3 format, a streamlined variant of Cornell’s QTIP method, to compress models down to as little as 1.6 bits per weight while keeping them coherent. The library supports dynamic batching, speculative decoding, multimodal inputs, and LoRA adapters across a sprawling list of architectures from Llama to Qwen.

The interesting bit

The quantizer computes Hessians on the fly and uses a fused Viterbi kernel to convert models in a single pass—minutes for small models, a few hours for 70B parameters on a single RTX 4090. That is a sharp contrast with methods like AQLM that demand server-grade hardware and nearly a thousand GPU-hours. The converted models also keep their original tensor names and structure, so they are not forcibly shoehorned into a Llama-shaped schema like EXL2 did.

Key highlights

• EXL3 quantization based on QTIP; supports 2–8 bit cache quantization and flexible bitrates
• Tensor-parallel and expert-parallel inference aimed at consumer multi-GPU setups
• Broad architecture support including multimodal models (Gemma 3/4, Qwen-VL, Mistral 3, etc.)
• Marlin-inspired GEMM kernel that reaches roughly memory-bound latency at 4 bpw on an RTX 4090
• OpenAI-compatible serving through the recommended TabbyAPI companion project

Caveats

• ROCm support is on the to-do list, so AMD GPUs are out for now.
• The GEMM kernel still needs work to match the same efficiency on Ampere GPUs and at lower bitrates.
• Gemma 4 currently lacks tensor/expert parallelism support, and the authors warn that some features may still be “a little broken at first.”

Verdict

Worth a look if you run local LLMs on consumer NVIDIA hardware and want to avoid cloud quantization bills. Skip it if you are on AMD ROCm or need a battle-tested, fully stable release today.