Treat LLMs like container images—because they are

What it does

llmman is a Rust CLI that wraps local inference servers—upstream llama.cpp, vLLM, SGLang, mlx-lm—and hosted providers into one control plane. A single launch command points agents like Claude Code, Codex, or Aider at a model, starts the correct backend, and execs the agent. It also exposes an OpenAI/Ollama/Anthropic-compatible HTTP endpoint for ordinary clients.

The interesting bit

Instead of inventing a custom blob format, llmman stores models as unmodified GGUF and safetensors files inside standard OCI Image Layouts. That means Docker Hub, GHCR, Harbor, or your internal registry can host models exactly like they host containers, and you can sign them with cosign. Several llmman serve nodes can also form a peer-to-peer aggregation: every node knows the others, and a request to any machine is served by whoever already has the model loaded—or the most free memory.

Key highlights

• One command launches an agent against a local or hosted model; no manual backend setup.
• Models are plain OCI artifacts: pull from Hugging Face, push to your own registry, or transfer directly between the two without touching your local disk.
• Uses upstream inference engines unmodified—no forked llama.cpp, no import step, no private blob format.
• Built-in cosign-compatible signing and per-repository trust policies for verification on every pull.
• Serves Ollama, OpenAI, and Anthropic APIs from one port, with on-demand loading and automatic unloading after idle time.
• Can pool multiple machines into an aggregation that looks like a single endpoint.

Caveats

• Pooling machines requires manually listing every peer on every node; there is no central scheduler or service discovery.
• Diffusion pipelines in Diffusers-layout need a separate vLLM-Omni install unless you run them inside a Docker runtime.
• The AGY integration requires version 1.1.13 or newer.

Verdict Worth a look if you manage models like you manage containers and want your existing registry, mirroring, and signing infrastructure to apply to weights. If you just need a single-machine chat UI and do not care about supply-chain standards, Ollama is already simpler.