Bare-bones inference for Mistral's full model catalog

What it does

mistral-inference is Mistral AI’s reference implementation for running its models on local hardware. It wraps model loading, tokenization, and generation into a small Python library and a pair of CLI tools, mistral-demo and mistral-chat. The package covers the full family of Mistral models—from the lightweight 7B to the massive 8x22B Mixture-of-Experts, plus vision and code variants like Pixtral and Codestral—offering both direct tarball downloads and Hugging Face Hub integration.

The interesting bit

The README calls the code “minimal,” and it means it: this is not a full-featured serving stack, but a thin, official wrapper that gets out of the way. That minimalism is the point—it acts as a stable reference for how Mistral expects its models to be loaded and tokenized. The library also handles multi-GPU sharding for the larger MoE models via torchrun, a non-trivial bit of orchestration tucked into an otherwise simple package.

Key highlights

• Supports the full Mistral catalog, including Mixtral MoEs, Codestral, Mathstral, Pixtral, and Mistral Large 2.
• Ships with mistral-demo and mistral-chat CLI tools for quick testing and interactive conversations.
• Python API exposes Transformer and generate classes for programmatic use.
• Multi-GPU execution for large models via torchrun without extra configuration.
• All listed models support function calling; some weights carry custom non-commercial licenses (MNPL/MRL).

Caveats

• Installation currently requires a GPU because the xformers dependency needs one to build or install, making CPU-only environments a non-starter.
• Several model links—notably the 8x7B variants—are marked “Updated model coming soon!” and may be stale.

Verdict

Grab this if you want an official, no-surprises baseline for running Mistral models locally or if you need a starting point before migrating to heavier production frameworks. Skip it if you need advanced serving features like continuous batching, quantization, or CPU inference—this is a reference runner, not a production engine.