Microsoft's 20 MB AI runtime for client apps that work offline

What it does

Foundry Local is an SDK and runtime for embedding local AI into desktop or client applications. It bundles ONNX Runtime, a curated catalog of quantized models, and automatic hardware detection into a roughly 20 MB package that runs inference in-process. Applications can perform chat completions or audio transcription entirely offline, with no API keys, cloud costs, or backend infrastructure.

The interesting bit

The project treats local inference as a shipping product rather than a science experiment. It exposes an OpenAI-compatible API so existing clients can redirect to a local endpoint with minimal changes, and it manages the full model lifecycle—downloading, caching, and selecting the best NPU, GPU, or CPU variant automatically. Microsoft is essentially packaging AI like a system library: bundled, offline, and invisible to the end user.

Key highlights

• ~20 MB runtime with native SDKs for C#, JavaScript, Python, and Rust
• Curated, quantized model catalog including Qwen, DeepSeek, Mistral, Phi, and Whisper
• Automatic hardware acceleration across NPU, GPU, and CPU via ONNX Runtime
• OpenAI-compatible request/response formats, including the Responses API
• Optional local server for prototyping, but core design favors in-process inference

Caveats

• Explicitly not designed for server or multi-user scenarios; no concurrent request batching
• Model catalog is intentionally limited and curated, so arbitrary community models are unsupported
• Windows hardware acceleration requires a separate WinML SDK; macOS and Linux use the standard package

Verdict

Worth evaluating if you are building desktop or client applications that need private, offline AI without backend infrastructure. Skip it if you need to serve many concurrent users or want the freedom to run any unvetted model.