CVPR Oral: A 3D Transformer You Can Copy-Paste Into Your Framework

What it does

PTv3 is a transformer backbone for 3D point cloud perception. It targets semantic segmentation across indoor benchmarks like ScanNet and S3DIS and outdoor lidar sets like nuScenes and Waymo. This repository hosts experiment records, configurations, and a standalone copy of the architecture that can be lifted into other codebases.

The interesting bit

The paper bills itself as “Simpler, Faster, Stronger” and earned a CVPR'24 Oral slot. Rather than locking the model inside a monolithic framework, the authors let you copy model.py and a serialization folder into your own project—an unusual concession in academic release culture. It also treats FlashAttention as a first-class dependency for speed, though you can disable it if your CUDA stack predates 11.6.

Key highlights

• CVPR'24 Oral selection (3.3% of accepted papers).
• Published validation mIoU scores include 77.6% on ScanNet, 80.3 on nuScenes, and 71.2 on Waymo.
• Two deployment modes: integrated into the Pointcept framework or extracted as standalone modules.
• Optional FlashAttention acceleration; without it, the model falls back to standard attention with smaller patch sizes.
• Companion project Sonata provides self-supervised pre-trained PTv3 weights and visual demos.

Caveats

• Released model weights are temporarily invalid because the model structure has been adjusted since the initial release.
• Several Model Zoo entries—notably SemanticKITTI and some PPT joint-training variants—lack published scores.
• Enabling FlashAttention forces fp16 accuracy and disables relative positional encoding; keeping those features means slower attention.

Verdict

Worth exploring if you want a modern point-cloud backbone and the flexibility to either adopt the full Pointcept ecosystem or just borrow the model code. Skip it for now if you need immediately stable pretrained weights for production—the current checkpoints are in flux.