Local agents that watch your screen so you can look away

What it does

Observer is a TypeScript app where you compose “micro-agents” from three parts: sensors (screen capture, OCR, camera, microphone, clipboard, audio), a small local LLM, and tools (notifications, memory, recording, even mouse clicks). Each agent runs on a loop interval, feeds the captured input to the model via prompt variables like $SCREEN or $ALL_AUDIO, and then executes a small JavaScript snippet you write against the model’s response — say, firing a Telegram message when a keyword shows up.

The interesting bit

The boring part is the good part: it’s a structured way to build “watch for X, then do Y” automations without writing a whole pipeline yourself. Agents also share state — getMemory/appendMemory tools let one agent read another’s memory by ID, so you can chain observers into something slightly smarter than a cron job with eyes.

Key highlights

• Runs fully local: bundled llama.cpp for any GGUF model, or any OpenAI-compatible endpoint (Ollama, vLLM) via a Custom Model Server URL
• Rich sensor set including screen OCR, camera OCR, clipboard, and mixed mic + screen-audio transcription (Whisper via transformers.js) for meetings
• Notification fan-out: Discord, Telegram, WhatsApp, SMS, email, Pushover, phone calls with TTS
• Agents can start/stop other agents and trigger mouse clicks — so it’s not purely passive
• Browser-based agents run sandboxed; there’s also a hosted web app and a shareable community library

Caveats

• Browser-only model loading (Transformers.js) is described by the author as unstable and crashes mobile devices; the desktop app is the stable path
• SMS, calls, and WhatsApp require whitelisting your number by contacting the developer, and SMS is blocked for US/Canada due to A2P policy
• Docker and Jupyter setups are marked deprecated

Verdict

Worth a look if you want privacy-first screen-watching automations — meeting transcription alerts, “tell me when this page changes” style tasks — without shipping your screen to a cloud API. Skip it if the idea of an LLM with a click() tool and your camera makes you nervous; that’s a reasonable reaction.