CatchMe: Vectorless long-term memory for desktop AI agents

What it does CatchMe is a local activity-capture layer for AI agents. It runs background recorders to monitor window focus, keystrokes, mouse movement, screenshots, clipboard, and notifications, then structures that stream into a hierarchical activity tree spanning Day, Session, App, Location, and Action. Each node gets LLM-generated summaries, turning raw logs into a browsable knowledge base that agents query through the CLI. There is also a web dashboard for interactive timelines and natural-language chat over your stored history.

The interesting bit The project skips vector embeddings entirely. Instead of stuffing screenshots into a database and hoping cosine similarity finds the right one, CatchMe keeps everything in a tree and lets an LLM navigate it top-down—reading summaries, picking branches, and drilling into raw evidence only when needed. It runs at roughly 0.2 GB RAM using SQLite and FTS5, and can operate fully offline with local LLMs such as Ollama or vLLM.

Key highlights

• Vectorless retrieval: Searches a hierarchical activity tree rather than using embeddings or vector databases.
• Privacy-first storage: All raw data stays local in SQLite; supports fully offline operation via Ollama, vLLM, or LM Studio.
• Agent integration: Ships as a single skill file for CLI agents including OpenClaw, NanoBot, Claude, and Cursor.
• Multimodal requirement: The LLM must handle both text and images, since retrieval inspects screenshots as evidence.
• Web UI: Offers interactive timelines, memory tree navigation, and natural-language chat beyond the CLI.

Caveats

• Cloud exposure: If you do not use a local model, your daily activity summaries—and the raw screenshots and keystrokes that feed them—are sent to whichever cloud API you configure.
• Model specs matter: The LLM needs a context window large enough to handle the configured max_tokens limits during summarization and retrieval, which may exclude smaller or cheaper models.

Verdict Developers who want their CLI agents to carry persistent, contextual memory of actual work should look here—especially if they prefer to avoid vector databases and cloud data exposure. If the idea of a background process logging your keystrokes and screenshots feels invasive, this is not your project.