Teaching LLMs to see in 3D without rendering a pixel

What it does

PointLLM is a multimodal LLM that ingests colored point clouds—xyz coordinates plus RGB values—and converses about the objects they represent. A point encoder projects 8192-point cloud features into the token space of a Vicuna backbone, which then processes interleaved point and text tokens to generate responses. The project ships training code, evaluation scripts, and pretrained 7B/13B checkpoints for tasks like 3D object classification and captioning.

The interesting bit

Instead of rendering point clouds to 2D images and bolting on a vision encoder, the team treats raw point clouds as a direct modality for the LLM. They also built a 730K-sample instruction dataset from Objaverse—660K simple descriptions plus 70K complex instructions generated by GPT-4—and had to invent new benchmarks because standard NLP metrics poorly judge 3D semantic accuracy.

Key highlights

• Consumes raw colored point clouds ((8192, 6) numpy arrays) via a dedicated point encoder mapped to LLM latent space
• Trained on 730K point-text instruction pairs from Objaverse using a two-stage strategy
• Evaluated with custom benchmarks for generative 3D classification and captioning, including GPT-4-based judgment
• Available in 7B and 13B variants based on Vicuna-v1.1
• ECCV 2024 best-paper candidate; improved PointLLM-V2 accepted at TPAMI 2025

Caveats

• The online demo was closed in April 2025; there is no hosted interface
• The authors caution that traditional NLP metrics (BLEU-1, ROUGE-L, METEOR) poorly capture semantic accuracy for 3D tasks
• Reproducing results requires downloading ~77GB of Objaverse point cloud data and multiple annotation files

Verdict Grab this if you’re researching 3D vision-language models or need a baseline for point-cloud understanding beyond rendered views. Skip it if you want a hosted API or a plug-and-play chatbot—the demo is offline and the data footprint is substantial.