An early open-source agent that lets LLMs use your desktop

What it does

The Self-Operating Computer Framework treats a multimodal LLM as a human operator: it feeds the model screenshots of your desktop, and the model replies with mouse and keyboard actions to reach an objective. Born in November 2023, it was among the first open-source projects to attempt full computer-use control via vision alone.

The interesting bit

Rather than asking the model to guess pixel coordinates in a raw screenshot, the default mode runs OCR to generate a hash map of clickable elements. The LLM reasons about labeled buttons and links, and the framework translates those choices back into precise clicks. A Set-of-Mark mode using a bundled YOLOv8 model is also available, though the README notes that OCR currently outperforms both it and vanilla vision prompting.

Key highlights

• Swappable model backends: GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL, and local LLaVA through Ollama
• OCR-based element grounding is the default; vanilla vision and YOLOv8 Set-of-Mark prompting are supported alternatives
• Runs on macOS, Windows, and Linux with an X server
• Optional voice input for stating objectives aloud
• One of the first open-source examples of full computer-use, dating back to November 2023

Caveats

• The default GPT-4o pipeline requires an OpenAI account with at least $5 in lifetime API spend to unlock rate-limited access
• Local LLaVA error rates are described as “very high”; the README positions it as a baseline for future local models rather than a practical driver
• macOS users must manually grant Terminal screen-recording and accessibility permissions

Verdict

Worth a look for researchers and developers building autonomous desktop agents or benchmarking visual grounding techniques. If you need reliable, unsupervised automation, the README itself flags that local model support is currently more of a baseline than a practical driver.