Inference · Serving

FractalCore schedules training tasks across mobile nodes, averages their checkpoints with FedAvg, and pays them in "Liquid MBs" — plus it slices Llama-class models into phone-sized pieces.

Project Maya runs GLM-5.3-Flash privately on one to sixteen GPUs by keeping hot experts in VRAM, warm ones in RAM, and the rest on NVMe.

AnyJev turns any open LLM into a calibrated decision engine—typed choices with probabilities you can threshold—without touching a single model weight.

It packages a C++17 video analytics runtime, a browser-based pipeline editor, and async VLM nodes into a single deployable appliance stack for edge hardware.

A C++ inference stack that squeezes ten large models — up to 114 GB — onto consumer PCs and serves them through a localhost OpenAI/Anthropic API.

ollaya applies Ollama's pull-and-serve ergonomics to decision models — small networks that return calibrated probabilities in a single forward pass and never generate a token.

PS5LM runs llama.cpp on a jailbroken PS5's GPU and serves any GGUF model to your whole network — no PC, no cloud, no account.

StartLux-Decision treats decisions as classification, not chat: one forward pass, a calibrated probability per option, and reasoning only when confidence dips below a gate.

TensorFold serves a short list of open model families on Apple Silicon and NVIDIA GPUs behind an OpenAI-compatible API, and accepts a drafted token only when it equals what the same engine would emit serially.

YuE2 Studio runs the open YuE2 song model locally on Windows and returns an editable score alongside the vocals — so you can change the performance without recomposing.

It rewrites every kernel for AMD Strix Halo and a single 179B-parameter MoE family to beat general-purpose engines without spending fewer bits.

Laya runs in Node.js via ONNX Runtime, turning tickets and emails into calibrated choices, rubric scores, and P(true) values in a single forward pass.

A Mac-native inference engine for coding agents that buys 2.5–5.3× decode speed over llama.cpp by being deliberately per-model.

opencode2api fronts OpenCode Zen and Zen Go with Chat Completions, Responses, and Anthropic Messages endpoints, plus key pools, proxy rotation, and a built-in WebUI.

SparkDiffusion stacks sparse attention, low-rank approximation, and distillation to claim 200×+ faster video inference on Wan models.

TypeLLM forces existing autoregressive LLMs to emit JSON Schema-valid outputs—enums, booleans, bounded numbers, and strings—without retraining or architectural changes.

KVMem parks long-context KV history in host RAM (and optionally SSD), paging only the relevant slices back onto the GPU per query.

FreeVideo brings the MiniMax H3 video model to consumer GPUs by streaming weights and picking the right kernels for whatever card you happen to own.

Gufo ditches portability on purpose: a C++ inference engine built solely for AMD's Strix Halo APU, where 128 GiB of unified memory makes a dedicated engine worth writing.

A local gateway for Claude Code, Codex and a dozen other AI clients: swap upstreams without touching client configs, keep secrets away from relays, and stop tool calls a relay smuggles into an answer.