Prototype LLM scheduling without burning cloud credits

What it does Vidur models how large language model serving systems behave under load. You feed it a workload trace or synthetic requests, pick a model and hardware layout, and it predicts metrics like time-to-first-token, per-token latency, and batch size over time. The goal is to let you compare deployment strategies—say, different tensor-parallelism splits or scheduling policies—without actually provisioning the cluster.

The interesting bit The twist is that once you complete an initial GPU profiling phase to capture execution kernels, the rest of the simulation runs entirely offline. That means you can iterate on capacity planning or try out speculative decoding variants on a laptop, using only the captured performance model to drive the results.

Key highlights

• Supports concrete models including Llama 2/3, CodeLlama, Qwen-72B, and InternLM-20B against specific hardware profiles like A100 and H100 DGX nodes.
• Validates scheduling ideas such as sarathi and other approaches using real-world Azure trace data or synthetic Poisson workloads.
• Exports detailed Chrome traces and standard inference metrics (TTFT, TPOT, end-to-end request time) for post-hoc analysis.
• Allows mixing tensor and pipeline parallelism dimensions to simulate multi-GPU node layouts.

Caveats

• The supported model/hardware matrix is sparse; several combinations like Llama 3 on H100 are explicitly unsupported.
• Expanding context length beyond 4k tokens requires passing additional predictor configuration flags.
• Several newer features such as prefix caching live on a canary branch that the authors warn has “sharp edges.”

Verdict Worth a look if you are designing LLM serving infrastructure or publishing scheduling research and need reproducible baselines with only a brief initial GPU profiling phase. Less useful if you simply want to profile a single real model instance.