Teaching GPT-4V to Browse Like a Keyboard Nerd

What it does vimGPT feeds GPT-4V a live screenshot stream from a Playwright browser running the Vimium extension. The model issues keyboard commands—triggering Vimium’s link hints and shortcuts—to navigate, click, and scroll without receiving any raw HTML or DOM text. You can also state an objective aloud in voice mode and watch the agent attempt to carry it out in real time.

The interesting bit The project treats Vimium’s yellow keyboard labels as a visual API for the page. Since GPT-4V can read on-screen letters but struggles with precise pixel coordinates, the model simply “presses” the key shown over a link to interact with it. It is a low-tech, clever workaround for the lack of native vision function calling.

Key highlights

• Purely vision-driven: the model sees the page through screenshots, not markup.
• Uses Vimium overlays as spatial shortcuts, avoiding coordinate-guessing.
• Voice mode allows hands-free objective setting.
• Runs a local Playwright browser with a manually loaded Vimium extension.
• Referenced in VisualWebArena research and WIRED coverage.

Caveats

• The Vision API lacks JSON mode and function calling, forcing “more primitive prompting methods.”
• Low-resolution screenshots cause the model to miss elements entirely.
• It currently spins up an artificial browser; it cannot yet control your personal profile or saved passwords.

Verdict A neat sandbox for anyone building multimodal web agents or probing the limits of vision-only interaction. If you need a stable daily driver for actual browsing tasks, this is still firmly in the experiment column.