What it does

BrickGPT takes a text prompt like “table with four evenly spaced legs” and outputs a buildable brick structure in standard LDraw format. It is a Llama-3.2-1B model fine-tuned on a custom dataset, with a post-generation stability check that rejects bricks that would collapse or collide. The repo includes inference, texturing, mesh-to-brick conversion, and fine-tuning pipelines.

The interesting bit

The model doesn’t just hallucinate a shape and hope for the best. It generates brick-by-brick coordinates, then runs a rejection loop using Gurobi for physics-based stability analysis (or a simpler connectivity fallback). The README shows a real trace: 59 bricks accepted, 98 rejected, 4 regenerations. It is slow, but it is honest.

Key highlights

• Fine-tuned from Llama-3.2-1B-Instruct on the StableText2Brick dataset
• Outputs LDraw (.ldr), rendered PNG, and plain-text brick lists
• Optional Gurobi-based physics validation; falls back to connectivity check without a license
• Additional pipelines for UV texturing and converting existing 3D meshes to bricks
• ICCV 2025 Best Paper (Marr Prize)

Caveats

• Requires gated Hugging Face access for the base Llama model and a separate Gurobi license for full physics checks
• Rendering needs ImportLDraw, a background EXR file from Google Drive, and the LDraw parts library in your home directory
• Inference takes ~60 seconds per prompt on the example hardware shown

Verdict

Grab this if you are doing generative 3D, robotic assembly, or just want to see an LLM constrained by actual Newtonian mechanics. Skip it if you need real-time generation or lack patience for academic software setup.