DETRs finally outrun YOLO on the speed-accuracy treadmill

What it does RT-DETR is a real-time object detector built on the DETR transformer architecture rather than the convolutional backbone YOLO relies on. The repo hosts official implementations of two CVPR-tier papers—RT-DETR and RT-DETRv2—with both Paddle and PyTorch code, pretrained weights, and a family of scaled models from R18 to R101.

The interesting bit The central bet is that transformers can escape the usual speed-accuracy tradeoff that favors CNNs. The authors claim their smallest model hits 217 FPS on a T4 with TensorRT FP16 while pushing mAP higher than comparable YOLO variants. RT-DETRv2 then squeezes further gains through “bag-of-freebies” training tweaks—no architecture surgery required.

Key highlights

• Dual-framework support: native Paddle releases plus a PyTorch port with converted weights and hubconf.py for torch.hub loading
• Scaled model zoo: six sizes, with the R18 running at 217 FPS and R101 topping out at 56.2 mAP on Objects365-then-COCO
• Deployment footprints: ONNX Runtime, TensorRT, and OpenVINO discussed in issues; also wrapped into Hugging Face Transformers and Ultralytics
• Sliced inference added for small-object detection; RegNet and DLA34 backbones now available alongside ResNet
• RT-DETRv4 announced as a separate repo using vision foundation models, though this repo stays focused on v1/v2

Caveats

• The speed claims are T4 TensorRT FP16 numbers; your mileage will vary on other hardware and without optimization
• README is heavy on benchmark tables and light on architecture explanation—expect to read the papers for design rationale
• PyTorch weights were originally converted from Paddle, so subtle numerical mismatches are possible

Verdict Worth a look if you’re locked into real-time detection but curious whether transformers have matured past their sluggish reputation. Stick with YOLO if you need battle-tested community support and one-command deployment; switch here if you want CVPR-credentialed numbers and don’t mind reading papers to debug.