A typed Rust leash for your local LLM

What it does

ollama-rs is an async HTTP client that maps Ollama’s REST endpoints into Rust structs and methods. It covers the full surface: text generation (with streaming and options), chat with persistent history, embeddings, model management (create, copy, delete), and even function calling. If you’re running Ollama locally, this library handles the wire protocol so your Rust code deals in ChatMessage and GenerationRequest instead of raw JSON.

The interesting bit

The standout is the Coordinator type, which runs an agent-like loop: it prompts the model, parses tool requests, executes Rust implementations like DDGSearcher or Calculator, and feeds the results back to the LLM. The #[ollama_rs::function] macro is the neat trick—it turns an ordinary async Rust function and its doc comments into a tool the coordinator can invoke, using your documentation as the LLM’s instruction manual.

Key highlights

• Full API coverage: generation, chat, embeddings, and model CRUD, including streaming behind feature flags.
• Built-in chat history via the ChatHistory trait; pass a Vec<ChatMessage> or roll your own storage.
• Coordinator orchestrates multi-step tool use, wiring LLM outputs to concrete Rust implementations and looping the results back in.
• The function procedural macro converts async Rust functions into LLM-callable tools, deriving descriptions from doc comments.
• Supports reasoning/thinking modes for models that expose them.

Caveats

• The README warns that the master branch may contain breaking changes; stick to released crates if you want stability.
• The provided examples explicitly use “poor error handling for simplicity,” so treat them as API sketches, not production templates.

Verdict

Worth a look if you’re building local-first LLM apps or async Rust agents that need structured tool use. If you’re just making the occasional curl call to Ollama, it’s probably overkill.