Inference · Serving

A 125-billion-parameter model that normally needs a server, packed onto one gaming PC — NVIDIA card plus RAM plus SSD — writing answers faster than you can read.

OmniRoute keeps your coding agents online by automatically failing over across 177 AI providers—including free tiers—when quotas run dry.

It exists because clicking 'generate' isn't enough when you need to control every model, parameter, and preprocessing step.

vLLM is an open-source inference engine that pages attention key-value memory like an operating system to drive higher GPU throughput, then exposes it through an OpenAI-compatible API.

It aggregates the free tiers of sixteen LLM providers into one OpenAI-compatible endpoint so you can experiment without juggling rate limits.

It breaks the vendor lock on Codex and Claude Code by translating their API calls to any LLM backend you choose.

SGLang exists to push low-latency, high-throughput inference for LLMs and multimodal models from a single GPU up to massive clusters.

VoxCPM2 proves TTS doesn't need discrete tokens: a 2B-parameter diffusion model generates continuous 48kHz speech for 30 languages and text-prompted voice cloning.

oMLX exists because running local LLMs on a Mac shouldn't mean choosing between a slick GUI and granular control over memory and caching.

It turns DeepSeek Harness’s command-line Web UI into a native macOS and Windows app so you can skip Node.js setup entirely.

LocalAI wraps 36+ inference engines behind one OpenAI-compatible API and pulls them on demand, so you can run LLMs, vision, voice, and video on anything from a CPU to a Jetson.

A TTS project built to clone realistic voices from just one minute of training audio.

Open-source infrastructure for building, benchmarking, and deploying AI agents that control real macOS, Linux, and Windows desktops.

It exists so you can download, run, and chat with open-weight LLMs locally through one CLI and REST API, keeping inference on your own silicon.

Upscayl gives desktop users a free way to enlarge and enhance low-resolution images using local Real-ESRGAN models and a Vulkan-compatible GPU.

Modular open-sourced its entire AI stack to let you serve models and write GPU kernels across hardware without switching tools.

Because juggling native APIs from a dozen LLM vendors, each with its own auth and billing, is a recipe for migraines.

A reimplementation of OpenAI's Whisper that trades the original inference engine for CTranslate2 and gains up to 4× speed without sacrificing accuracy.

It centralizes model definitions so the same architecture works across PyTorch, JAX, vLLM, and llama.cpp without rewrites.

Qdrant stores neural network outputs as searchable vectors and lets you filter them with SQL-like payload queries, bridging the gap between embedding models and production search.