A chat client that runs GGUF models on your phone, no cloud required

What it does

PrivateLM is a cross-platform AI chat client built in Flutter. On Android and iOS it runs GGUF models locally via a custom llama.cpp wrapper, with Vulkan offload on Android and Metal on iOS. When local hardware isn’t enough — or you’re on the web build — it falls back to OpenAI, Anthropic, Gemini, or Kimi through a single normalized interface. Chats, tasks, and settings persist locally in Hive; nothing leaves the device unless you pick a cloud provider.

The interesting bit

The app profiles your device at first launch — RAM, GPU tier, thread count — and auto-recommends context size and token limits, bucketing phones into ultra/high/mid/low tiers. It also does on-device image generation, with the README showing results from a Pixel 6A and a Poco F1, which is a fun flex for a phone app.

Key highlights

• Local GGUF inference with Vulkan (Android) and Metal (iOS) GPU acceleration; fully offline after model download
• One CloudService normalizing four different API shapes, including Kimi’s OpenAI-compatible endpoint
• Multimodal chat with vision support on both local models (Qwen2-VL) and cloud providers
• Native chat-template support (ChatML, Llama-3, Gemma, Phi) with manual prompt fallback
• Idle detection and hard timeouts keep generation from hanging underpowered hardware

Caveats

• Web is cloud-only; local inference there is “coming soon” with no timeline given
• iOS distribution is a sideloaded IPA via AltStore, which expires every 7 days — iPad is the recommended target due to RAM needs
• The repo description mentions “uncensored” models, but the README doesn’t document which models are actually recommended or tested beyond Qwen2-VL

Verdict

Worth a look if you want a genuinely offline LLM client on Android with sensible device-aware defaults. Skip it if you need a desktop app or anything other than sideloading on iOS.