Tracking objects without playing frame-to-frame matchmaker

What it does MOTR is an end-to-end multi-object tracker built on Deformable DETR. Instead of detecting objects per frame and then matching them across time with spatial or appearance heuristics, it uses persistent “track queries” that model entire object trajectories and update frame by frame. The network ingests video and outputs complete tracks without a separate association stage.

The interesting bit Each track query follows one object through the whole sequence, learning long-range temporal variation implicitly rather than relying on post-hoc matching heuristics. A temporal aggregation network and tracklet-aware label assignment attempt to keep queries from drifting as objects occlude or change shape.

Key highlights

• Fully end-to-end: no explicit detect-then-associate pipeline
• Built on Deformable DETR with ResNet50 backbones, COCO pretrained
• Benchmarked on MOT17, DanceTrack, and BDD100K
• Inference runs at roughly 7.5 FPS on 1536×800 video
• Training requires 8 GPUs and takes 2.5 days on V100s (4 days on RTX 2080 Ti)

Caveats

• Training is resource-heavy: 8 GPUs required, and convergence takes days even on high-end hardware
• Inference at ~7.5 FPS is far from real-time for high-resolution video
• The codebase is essentially a heavy extension of Deformable DETR, not a lightweight standalone tool

Verdict Worth exploring for researchers building transformer-based trackers or anyone needing a strong ECCV 2022 baseline. Production engineers hunting for real-time, edge-friendly trackers should keep scrolling.