An Android agent that sees your screen and taps for you

What it does

OpenOmniBot is a native Android agent built in Kotlin and Flutter that runs entirely on your device. It uses vision models to read the screen and physically operate apps, gestures, and system settings through Android’s accessibility services. The project wraps a full agent loop—understand, decide, execute, reflect—around local and remote models, giving the phone a memory system, scheduled tasks, and even an embedded Alpine environment and terminal.

The interesting bit

The architecture is deliberately modular: a Flutter UI layer, a native automation engine, an accessibility and screen-perception service, and a local inference runtime that can be stripped out for a slimmer build. It also exposes a skill marketplace model where you install capabilities by feeding the bot a GitHub repository link.

Key highlights

• On-device operation with vision-based UI automation (taps, gestures, app control)
• Dual build targets: a standard slim edition or a full omniinfer edition with local LLM inference via MNN or llama.cpp
• Extensible skill system installable via repository links
• Built-in memory with short-term, long-term, and embedding support
• Remote Codex bridge for pairing with a desktop CLI instance over LAN

Caveats

• Memory embedding requires a separate embedding model, and the README warns that most scenarios need multimodal or vision-capable models to work well.

Verdict

Worth a look if you want an open, hackable Android automation framework that stays on-device. Skip it if you just need a chat wrapper—this is closer to a mobile OS agent than a conversational UI.