Mesh your Macs into a local AI cluster

What it does

exo is a distributed inference engine that pools your local devices into a single cluster. It auto-discovers peers, shards large models across them using tensor parallelism, and serves results through OpenAI-, Claude-, and Ollama-compatible APIs. A built-in dashboard handles cluster management and chat without extra tooling.

The interesting bit

The project treats Thunderbolt 5 like a datacenter interconnect by shipping day-zero support for RDMA over Thunderbolt, which the team claims reduces inter-device latency by 99%. A topology-aware scheduler then splits models based on realtime link bandwidth and per-node resources rather than assuming a uniform grid.

Key highlights

• Zero-config mesh: devices find each other automatically with no manual cluster setup.
• RDMA over Thunderbolt 5 on macOS 26.2+, targeting sub-millisecond latency between Macs.
• Tensor parallelism with reported speedups up to 1.8× on two devices and 3.2× on four.
• Drop-in API compatibility with OpenAI Chat Completions, Claude Messages, and Ollama endpoints.
• Built-in web dashboard for cluster monitoring and inference.

Caveats

• Linux support is currently CPU-only; GPU acceleration is marked as under development.
• RDMA requires macOS Tahoe 26.2 or later and Thunderbolt 5 hardware (M4 Pro/Max, M3 Ultra).
• The packaged macOS app also requires macOS Tahoe 26.2, and enabling RDMA demands a trip to Recovery mode.

Verdict

Worth a look if you have a stack of Apple Silicon Macs and want to run 400B+ parameter models locally without cloud GPUs. Skip it for now if you are on Linux and need GPU performance, or if your idea of a cluster is a single laptop.