Vision-as-inverse-graphics, implemented as an LLM arguing with itself

What it does VIGA is a programmatic visual-reconstruction agent. Feed it a target image or task and it writes Blender Python or slide-layout code, renders the result, compares it to the target from multiple viewpoints, and revises the code if anything looks wrong. The loop repeats until the scene converges, handling static 3D scenes, dynamic physics-based animations, and 2D document layouts without finetuning the underlying model.

The interesting bit Instead of training a specialized vision model, VIGA treats graphics as code and perception as a unit test. A single LLM agent swaps between a Generator role that writes scene programs and a Verifier role that inspects renders for discrepancies, maintaining a memory of past attempts, code diffs, and feedback. It is essentially test-driven development for computer graphics.

Key highlights

• Supports multi-step 3D editing via BlenderBench, single-step graphics tasks via BlenderGym, and 2D slide synthesis via SlideBench
• Self-correcting loop requires no finetuning; the same agent plans, codes, renders, and verifies iteratively
• Verifier inspects output from multiple camera angles rather than trusting a single canonical view
• Maintains contextual memory across iterations, including plans, code diffs, and render history
• Produces real Blender Python scripts or PowerPoint layouts instead of static pixel predictions

Caveats

• Setup is heavy: the README calls for four separate Conda environments, API keys for OpenAI and Meshy, and manual path configuration
• The install instructions note you can “ignore the errors” during the Blender dependency step, which suggests brittle tooling
• 3D modes expect an NVIDIA GPU with CUDA support; without one, you are likely limited to 2D slide tasks

Verdict Researchers and graphics tinkerers who want an LLM that manipulates actual scene graphs rather than hallucinating pixels should look here. If you were hoping for a lightweight pip install and a one-line API call, this is not your project.