Inference · Serving

AirLLM slices giant transformers into layer shards so they fit in consumer VRAM without quantization or distillation.

It exists because clicking 'generate' isn't enough when you need to control every model, parameter, and preprocessing step.

SGLang exists to push low-latency, high-throughput inference for LLMs and multimodal models from a single GPU up to massive clusters.

A TTS project built to clone realistic voices from just one minute of training audio.

OpenBB normalizes proprietary and public financial data so engineers can feed the same sources to Python scripts, REST APIs, Excel, and AI agents without rebuilding integrations.

It wraps Stable Diffusion—and the whole messy ecosystem of models, upscalers, and LoRAs—into one browser-based workbench.

Ultralytics wants to stop you from stitching together separate repos for every computer vision task by bundling detection, segmentation, tracking, and pose estimation into one YOLO-backed package.

Modular open-sourced its entire AI stack to let you serve models and write GPU kernels across hardware without switching tools.

Qdrant stores neural network outputs as searchable vectors and lets you filter them with SQL-like payload queries, bridging the gap between embedding models and production search.

It exists to run large language models on virtually any hardware—from Apple Silicon to RISC-V to your browser—with zero external dependencies and minimal setup.

YOLOv5 made real-time object detection as easy as `torch.hub.load`, then exported to everything from iOS to edge chips.

Jan is a desktop chat client that makes running local LLMs as mundane as using ChatGPT, while quietly exposing an OpenAI-compatible API for your other tools.

It is a native mobile sandbox for downloading, benchmarking, and interacting with open-source LLMs and multimodal models entirely on-device.

DeepSeek-V3 exists to prove that a 671-billion-parameter model can train end-to-end without a single rollback, activate only 37B parameters per token, and still match leading closed-source systems.

Ray treats distributed computing as a Python primitive, then layers on libraries for training, tuning, serving, and reinforcement learning.

Because training a transformer shouldn't require 245MB of PyTorch just to multiply matrices.

Because swapping from GPT-4o to Claude shouldn't require rewriting your request plumbing.

LibreChat bundles every major LLM provider into a single self-hosted chat platform so teams don't have to choose—or leak data.

Microsoft built an inference engine that lets a single CPU run a 100B-parameter model at human reading speed by using 1.58-bit weights.

Mozilla wraps llama.cpp and a full model into a single cross-platform executable using an obscure libc trick.