Stop Teaching Agents to Click

Somewhere between the screenshot-reading GUI agents and the hand-built MCP servers that every SaaS vendor is now rushing to ship, there is a third position, and this month it acquired a very loud champion. CLI-Anything, out of the University of Hong Kong’s Data Intelligence Lab (HKUDS), crossed 40,000 GitHub stars in late May 2026 — by one count, nearly 4,800 of them in a single week. The pitch, printed right under the repo title, is blunt: today’s software serves humans; tomorrow’s users are agents; and the fastest way to bridge that gap is to stop teaching agents to use menus and start giving software a command line.

The project is two things stacked on top of each other. It is a methodology — a seven-phase pipeline, encoded in a playbook called HARNESS.md, that turns an arbitrary codebase into a tested, pip-installable CLI harness. And it is a distribution ecosystem — CLI-Hub, a registry at clianything.cc that currently lists around a hundred CLIs across 35 categories, mixing the project’s own generated harnesses with community-contributed and third-party public tools. Neither half is glamorous. That, arguably, is the point.

The argument underneath the hype

The intellectual case was laid out in a June 2026 paper by Yuhao Yang, Tianyu Fan, and Chao Huang, and it is more pointed than the README’s cheerleading suggests. The dominant approach to computer-use agents — the paradigm behind most “agent operates your desktop” demos — has agents interpret screenshots, locate UI elements, and execute mouse clicks, mimicking human interaction. The paper’s claim is that this paradigm is not merely inefficient but fundamentally misaligned with what agents are good at.

GUI agents, the authors argue, force models to emulate human perceptual limitations: brittle pixel-level interactions, timing dependencies, coordinate-based actions that break the moment an interface moves a button six pixels to the left. Agents, by contrast, are excellent at exactly the things GUIs hide — structured data processing, programmatic control, deterministic execution. The lossy translation from pixels back to computational state is a tax paid on every single action.

The alternative on offer: an agent-facing contract that sits above the real application backend. The harness generates valid project files — ODF for LibreOffice, MLT XML for Shotcut and Kdenlive, SVG for Inkscape — and then delegates final truth to the actual application for rendering. The paper is explicit that this is a contract, not a reimplementation: the CLI must call the real software, no Pillow stand-ins for GIMP, no custom renderers for Blender. Tests fail rather than skip when backends are missing, which is a design decision that quietly says a lot about how the team thinks about honesty in agent tooling.

This positions CLI-Anything in a debate that has been simmering all year. The agent-experience crowd has been making a related argument from the vendor side: that software now needs both a UX and an “AX,” and that most companies are cutting corners by relabeling their developer CLIs as agent-ready when the output was still designed for human eyes. HubSpot, notably, launched an agent-first CLI in private beta around the same time CLI-Anything was trending — evidence that the “agents love the command line” thesis has escaped academia and is now a product strategy. CLI-Anything’s twist is that it doesn’t wait for vendors to come around. If the software has source code, the pipeline builds the agent-native surface itself.

The boring part, which is where the value is

Strip away the vision and CLI-Anything is, at its core, a very disciplined prompt-and-verify system. The seven phases read like a software engineering checklist an agent is forced to march through: analyze the source and map GUI actions to the APIs underneath them; design command groups and state models; implement a Click-based CLI with a REPL, JSON output, and undo/redo; plan tests; write tests; document results; publish. A half-step between documentation and publishing generates a SKILL.md file — a machine-readable description of the CLI’s command groups and constraints — so that skill-discovery systems can index the harness and other agents can find and use it autonomously.

What makes this more than glue is HARNESS.md, the standard operating procedure that the pipeline enforces, and which reads like a distillation of real production scars. The “Rendering Gap” lesson: GUI applications apply effects at render time, so a CLI that manipulates project files but exports with a naive tool will silently drop effects — the fix is native renderer invocation, filter translation, then render script. Timecode precision: non-integer frame rates like 29.97 cause cumulative rounding, so the playbook mandates rounding over truncation and a tolerance of one frame in tests. Output verification: never trust a zero exit code — check magic bytes, ZIP structure, pixel analysis, audio RMS levels. These are the unglamorous details that separate a harness that produces a real PDF from one that produces a file with a .pdf extension, and the fact that they’re codified rather than tribal knowledge is a genuine contribution.

The evidence base is unusually concrete for this genre of project. The repository ships pre-built harnesses for dozens of applications — GIMP, Blender, Inkscape, LibreOffice, OBS Studio, Audacity, Kdenlive, Shotcut, Zotero, Calibre, Joplin, QGIS, FreeCAD, Godot, RenderDoc, even Slay the Spire II — with the README reporting 2,461 passing tests at a claimed 100% pass rate, split across unit tests, end-to-end tests against real software, and subprocess checks of installed commands. (Third-party coverage from May 24 cited 2,330; the number has been climbing as harnesses merge, which at least suggests the tests are attached to something that moves.) The demos are similarly artifact-forward: an agent assembling a Curiosity-style rover in FreeCAD under a live preview loop, a Blender orbital drone grown stage by stage with a replayable command trajectory, an HTTPS handshake diagram built entirely through Draw.io commands. The trajectory-and-preview pattern — every command tied to a matching visual state — is the project’s answer to the observability problem that plagues GUI agents, where you can’t tell what the agent did until you scroll a video.

The ecosystem play

The second half of the project is the more strategically interesting one. CLI-Hub is a pip-installable package manager plus a web registry, and it has been absorbing community contributions at a pace the news log makes visible: ArcGIS Pro cartography via an MCP bridge into a live Esri session, Joplin with a 134-test baseline, Rekordbox with guarded SQLCipher write paths, Calibre with real-application end-to-end evidence, an Obsidian CLI, a Hermes orchestration skill. A meta-skill lets agents discover and install harnesses themselves — the stated prompt pattern is simply asking an agent to find appropriate CLI software in the hub and complete a task, and letting it do the rest.

This is where CLI-Anything stops being a research artifact and starts being infrastructure with network effects. Every merged harness makes the registry more useful; every registry user has less reason to build their own wrapper. The project has also been aggressive about meeting agents where they live — plugins or skills for Claude Code, Codex, OpenCode, Pi, OpenClaw, Qodercli, and GitHub Copilot CLI, with the same seven-phase methodology adapted to each platform’s invocation style. The bet, implicitly, is that the skills ecosystem (ClawHub, SkillHub, and friends) becomes the distribution layer and CLI-Hub becomes the warehouse.

It is also, worth noting, a bet that runs adjacent to MCP rather than against it. The paper frames the three doors agents reach through as APIs, agent-native protocols, and CLIs — and the ArcGIS Pro harness happily wraps an MCP bridge when that’s what the target exposes. CLI-Anything’s claim isn’t that MCP is wrong; it’s that MCP coverage is thin, API-by-endpoint integration burns tokens, and a well-designed CLI can finish a whole job in one composable surface. The broader agent literature keeps landing on the same constraint from the other direction: agents improve when their work is grounded in environments that can execute, check, and return structured feedback. A tested CLI harness is precisely that.

The honest caveats

The project is refreshingly candid about its own limits, and the community coverage has been too. The README’s limitations section concedes three things that matter. First, harness generation depends on frontier-class models — the team names Claude Opus and Sonnet 4.6 and GPT-5.4 as the reliable tier — and weaker models produce incomplete or incorrect CLIs requiring significant manual correction. Second, the pipeline needs source code; compiled-binary targets degrade badly. Third, a single generation run rarely covers everything, and iterative refinement is often needed to reach production quality. A widely shared Facebook summary put the practical warning plainly: review and test generated interfaces before giving an agent access to important files.

The commit history adds its own quiet commentary. Security hardening passes have been frequent and non-trivial: a GIMP Script-Fu path injection fix, Sketch token-file handling hardened against path traversal and symlink escapes, untrusted XML parsing routed through defusedxml across a half-dozen formats, URL validation and DOM sanitization for the browser harness. None of this is damning — every large surface area project patches things — but it is a useful reminder that “agent-native” also means “a new attack surface with a language model holding the pen,” and that generated code deserves the same review bar as human code.

There is also a fair structural criticism the project mostly sidesteps: much of the value here is discipline, not invention. Click CLIs, JSON output flags, REPLs, and pip packaging are all old technology; the pipeline is a well-engineered automation of practices a strong engineer would apply by hand. But dismissing it as glue would miss the point — the entire history of infrastructure is making the disciplined thing the easy thing, and nobody had made “give your software an agent-grade CLI” a one-command, test-verified, registry-distributed operation before.

Where this goes

The roadmap items tell you where the team thinks the leverage is: a benchmark suite for agent task completion rates, harnesses for more categories (CAD, DAW, IDE, EDA), packaging closed-source software and web services into CLIs, and an experimental “CLI Matrix” — capability bundles mapping intents to provider CLIs, visible in early form on the hub site. The benchmark piece is the one to watch. If CLI-Anything can show that agents driving its harnesses complete real tasks at measurably higher rates than agents clicking through GUIs, the GUI-automation research agenda loses a lot of its air. If it can’t, the project remains a very good engineering convenience with an unproven thesis.

The macro context is on their side, at least directionally: Gartner’s forecast has agentic AI in a third of enterprise software by 2028, and IBM’s research has 80% of executives increasing agentic AI spend. Every one of those deployments will eventually hit the same wall CLI-Anything is trying to demolish: real professional software that speaks only pixels and menus. Whether the answer is this particular registry, vendor-built agent CLIs, or MCP servers that finally grow up, the argument has already shifted the conversation. The next time someone shows you an agent proudly clicking a button in Blender, the correct question is now: why is it clicking?