Ollama, but the models answer in probabilities instead of prose

What it does A single Rust binary runs a local daemon that pulls decision models by name and serves them over HTTP, with Ollama-familiar verbs (run, pull, list, ps, create) — the CLI even starts the daemon if it isn’t running. A decision model reads a state (an email, a ticket, any JSON) plus typed questions (choice, score, noul) and returns calibrated probabilities in one forward pass, in milliseconds. The output is numbers, not paragraphs. The HTTP layer is wire-identical to TypeSafe’s /v1/systemone, so existing Jev clients switch over by changing one environment variable.

The interesting bit The distribution model is the clever part: ollaya publishes only ~3 MB ONNX graphs per model, and those graphs read the original weight files straight from each author’s Hugging Face repository — pinned to a commit, verified by sha256. Weights are never re-hosted, and GGUF-based models run the author’s own file on llama.cpp. The parity claim is refreshingly concrete too: fp32 exports match the PyTorch reference on 100% of 2,383 questions per checkpoint.

Key highlights

• Speed: laya:en (ModernBERT-large, 421M) answers five questions in 8–10 ms on an RTX 4090; the recommended winnow:e4b takes 89 ms.
• A real zoo: NLI and zero-shot classifiers (nli, gliclass), a safety guard (qwen3guard), Mapika’s decider line, Jared Palmer’s calibrated kev, and a laya router that detects the language of each request.
• Modelfiles bake a question set into your own model, Ollama-style.
• Fully local once models are pulled: ONNX Runtime on CPU and CUDA, llama.cpp for GGUF (CPU, CUDA, Metal on Apple silicon).
• An mcp subcommand serves the models to Claude Code, Cursor, and other MCP clients, with a companion skill that teaches agents when to reach for them.

Caveats

• The recommended winnow:e4b is a 4B-class model; the README itself tells you to start with laya if you lack an NVIDIA GPU.
• Linux builds need glibc ≥ 2.38 (Ubuntu 24.04 or newer).
• Accuracy trails the hosted reference: 0.722 on typed decisions versus TypeSafe’s Jev at 0.738.

Verdict If you build triage, routing, or classification pipelines and want calibrated answers locally in milliseconds — especially if you already run Jev clients — this is worth a pull. If you need generated text or the last few points of accuracy, the hosted incumbents still win.