Rust AI toolkit that keeps LLMs on a short grammatical leash

What it does Kalosm bundles Rust crates for interacting with local pre-trained models across text, audio, and image modalities. It wraps quantized, GPU-accelerated variants of Llama, Mistral, Phi, Whisper, Segment Anything, and Bert in a unified interface. A companion runtime called Fusor—still in early development—is intended to become the default inference backend in the upcoming 0.5 release.

The interesting bit The standout feature is structured generation: a custom parser engine and sampler that constrains model output to arbitrary grammars, regex patterns, or native Rust types via derive macros. The project claims this structure-aware acceleration can run faster than uncontrolled text generation, turning the usual latency penalty of strict formatting on its head.

Key highlights

• Supports quantized, GPU-accelerated models including Llama (up to 70B), Mistral, Phi, Whisper, Segment Anything, and Bert embeddings.
• Structured generation enforces output formats like JSON, HTML, or XML through custom grammars and #[derive(Parse, Schema)] macros.
• Includes utilities for context extraction from txt/html/docx/md/pdf, semantic search with vector database integrations, audio transcription, and web scraping.
• Fusor uses a kernel fusion compiler to merge operation chains into optimized WGPU kernels without hand-written shader code, targeting browsers and AMD GPUs in the future.

Caveats

• Fusor is explicitly marked as early in development and not ready for production use; it will not become the Kalosm backend until the 0.5 release.

Verdict Worth a look if you are building Rust applications that need local, offline AI with strictly formatted outputs. Give it a pass if you need a production-ready, cross-platform GPU runtime immediately—Fusor is still experimental, and the current inference stack is left unspecified.