Teaching VLMs Depth Perception Without New Cameras

What it does

VQASynth ingests flat image datasets from Hugging Face and turns them into synthetic spatial Q&A pairs grounded in reconstructed 3D scenes. It detects objects, lifts them into metric point clouds, then generates questions about distances, orientations, and spatial relationships. The result is training data for vision-language models that need to reason about physical space instead of just describing it.

The interesting bit

The project is an open-source rebuild of the SpatialVLM pipeline, but it keeps the reconstruction stage current by swapping in newer specialist models—VGGT for metric depth, SAM2 for segmentation, and Florence-2 or Molmo for captioning. It also offers an agentic mode built on NVIDIA’s NOOA framework that abandons fixed templates and dynamically composes tool calls (detect_objects → metric_depth → distance_3d) based on the question at hand, letting it drop into robot-learning stacks like LeRobot as a grounded reward annotator.

Key highlights

• Reconstructs 3D scenes from single images using VGGT depth and point-cloud fusion, then templates spatial Q&A from the geometry.
• Ships with ready-made datasets (SpaceOm, SpaceThinker, OpenSpaces) and fine-tuned models benchmarked on SpatialScore, OmniSpatial, SpaCE-10, and MindCube.
• Exposes both a rigid batch pipeline for dataset synthesis and a dynamic NOOA agent that routes questions to the right tools on the fly.
• Persists full tool-call traces in OpenAI-messages format, making the chain of reasoning auditable and reusable for further VLM fine-tuning.
• Demands serious hardware: the authors specify an A10-class GPU with 24 GB VRAM and an NVIDIA container stack to run locally.

Caveats

• The hardware floor is steep; 24 GB VRAM and an NVIDIA container stack are mandatory, ruling out casual laptops.
• The README is upfront about the VRAM requirement but vague on exact throughput, cost-per-sample, or how the templated prompts generalize beyond the provided examples.

Verdict

Worth a look if you are building embodied-AI or robotics datasets and need to bootstrap spatial reasoning labels from existing 2D imagery. Skip it if you were hoping for a lightweight, CPU-friendly data augmenter.