An AI vision coder that plans, tests, and iterates—now retired

What it does VisionAgent is a Python library that takes a natural-language prompt plus an image or video and generates ready-to-run Python code for computer-vision tasks. It acts as an orchestration layer, selecting appropriate vision models from its own tool library—such as Florence2 for detection or SAM2 for tracking—and writing the glue code for you. The generated scripts are saved as standalone files you can run independently.

The interesting bit The agent doesn’t just emit code and hope; it writes a plan, generates the implementation, runs a test case, and loops back to fix errors if the test fails. That self-correction loop is unusual for a code-generation tool, and it treats vision tasks as software-engineering problems rather than one-shot inference jobs.

Key highlights

• Ships with a curated toolbox (vision_agent.tools) of standalone vision functions for detection, counting, tracking, and segmentation that can be imported directly into your own scripts.
• Uses Anthropic Claude 3.7 Sonnet and Google Gemini Flash 2.0 Experimental as planners and coders; you must bring your own API keys for both, plus a LandingAI API key.
• Supports video workflows, including frame extraction and object tracking across timestamps.
• Provides a local webapp in examples/chat for interactive prompting.
• Configurable LLM backends: swap the default models for OpenAI or others by editing vision_agent/configs/config.py.

Caveats

• The project is officially deprecated; LandingAI directs users to “Agentic Document Extraction” instead.
• Requires three separate API keys (LandingAI, Anthropic, and Google) to function, which means rate limits and billing are spread across multiple providers.
• The README is heavy on quickstart steps and light on architectural depth, so extending the agent beyond the provided examples may require reading the source.

Verdict Worth a look if you’re maintaining a legacy vision pipeline built on LandingAI’s tooling or studying agentic code-generation patterns, but new projects should follow the maintainer’s guidance and use Agentic Document Extraction instead. If you just need a one-off object detector, importing the underlying tools directly is probably overkill.