The crab and the llama call it quits

What it does

llm was a Rust ecosystem for loading and running quantized large language models through the GGML tensor library. It offered both a library crate and a CLI tool capable of REPL, chat, quantization, and perplexity calculations across several architectures. Bindings for Python and Node also existed.

The interesting bit

The project is now archived, and the maintainers turned the README into an unusually honest post-mortem: they enumerate exactly which branches are broken (the gguf branch only handles Llama, the develop re-port is abandoned) and provide a curated list of modern replacements. It functions less as software and more as a well-organized off-ramp.

Key highlights

• main branch supports GGMLv3 models, but not the newer GGUF format
• Model coverage included LLaMA variants, GPT-2, GPT-J, GPT-NeoX, BLOOM, and MPT
• Bundles a CLI with REPL, chat, quantization, and perplexity computation
• Inference was CPU-only; GPU backends were planned but never completed before archival
• Last crates.io release (0.1.1) is described by the authors as “very out of date”

Caveats

• Repository is archived and unmaintained due to lack of time and resources
• The gguf branch lacks quantization and only supports Llama models
• Even the main branch runs on an old GGML version and cannot load GGUF files

Verdict

Visit for the curated list of Rust LLM alternatives, not the code. If you actually need inference today, the maintainers explicitly recommend mistral.rs, Candle-based libraries, or llama.cpp wrappers instead.