Stable Diffusion on Apple Silicon, no cloud required

What it does The repository is a two-part bridge: a Python package that converts PyTorch Stable Diffusion weights into Core ML format, and a Swift package that drops the resulting models into Xcode projects. It targets the Neural Engine and GPU across macOS, iPadOS, and iOS, letting apps generate images without ever calling a cloud API. In short, it is Apple’s reference pipeline for running a datacenter-sized diffusion model on pocket hardware.

The interesting bit The benchmarks are unusually candid. Apple notes that the Swift wrapper still carries up to ~10% overhead unrelated to Core ML execution, and that the attention optimizations are generic to Transformers rather than tuned specifically for diffusion. The real craft is the quantization—squeezing weights down to six bits (and below for SDXL) while using just-in-time model loading to keep RAM-starved iPhones from crashing.

Key highlights

• Supports SD 2.1 (512×512) and SDXL (768×768 on mobile, 1024×1024 on Mac), with end-to-end latency as low as ~8 seconds on an iPhone 14 Pro Max.
• Applies aggressive Core ML weight compression—6-bit palettization for standard models, mixed-bit recipes dropping the SDXL UNet to roughly 4 bits.
• Offers both a Python diffusers-compatible pipeline for conversion and validation, and a native Swift package for app integration.
• Requires Apple Silicon hardware: M-series Macs or iPads, and A14-generation iPhones or later.
• Runs fully offline after the initial model conversion and bundling.

Caveats

• Swift inference carries up to ~10% overhead unrelated to the Core ML model execution itself, per Apple’s benchmark notes.
• On memory-constrained devices, the reduceMemory option loads and unloads models just-in-time to avoid crashes, adding up to two seconds—or “significant overhead” for SDXL—to generation time.
• The published performance numbers are explicitly not ceiling values; the optimizations are generic to Transformers and the Swift code remains un-tuned.

Verdict iOS and macOS developers who need private, offline image generation should evaluate this closely. If your stack lives on Linux or CUDA, there is nothing here for you.