An AI butler that screenshots your screen and talks back

What it does

macOSpilot is an Electron app that lurks in the background until a keyboard shortcut wakes it up. It grabs a screenshot of your active window, records a spoken question, and ships both to OpenAI’s Vision API. The answer pops up in a small floating overlay and is read aloud via OpenAI TTS, so you never have to leave the app you’re using.

The interesting bit

The project is essentially glue code: it wires macOS screen capture and microphone access to GPT-4 Vision and Whisper, turning any application into a context-aware help target. The author openly calls the code functional rather than elegant, which sets honest expectations.

Key highlights

• Application-agnostic: it screenshots whatever window is active, requiring no per-app integration.
• Answers are constrained to one sentence by default via the system prompt in index.js.
• Supports both voice input through Whisper and typed questions.
• Can be packaged as a standalone .app with Electron, though testing is limited to Apple Silicon and Intel Macs.
• Keeps a simple session history in a separate window you can minimize.

Caveats

• Your OpenAI API key is stored in the settings UI without encryption.
• Screenshot, audio, and TTS files are written to disk and overwritten each run, but never auto-deleted.
• Despite Electron’s cross-platform packaging scripts, the author has only tested on macOS.

Verdict

Try it if you want a hackable, always-on assistant that comments on whatever is currently on your screen. Look elsewhere if you need encrypted credentials, offline capability, or production-grade polish.