Turning video into 4D trajectory soup, one forward pass at a time

What it does

Trace Anything takes a video, an image pair, or even an unordered set of images and, in a single forward pass, spits out a “trajectory field” — a continuous, parametric 3D path for every pixel across frames. No per-scene training, no lengthy optimization loops. The output is a PyTorch tensor of control points, confidence maps, and foreground masks that you can poke at in an interactive 3D viewer built on Viser.

The interesting bit

The trajectory field representation itself: instead of treating video as a stack of 2D frames or a static point cloud with timestamps, it parameterizes pixel motion through time as smooth 3D curves. The model also predicts a scalar time ∈ [0, 1) for each view even when the true order is already known — a slightly superfluous head that the authors note anyway.

Key highlights

• Single forward pass inference; no scene-specific optimization
• Handles videos, image pairs, or unstructured image sets (though unordered support is flagged as future work)
• Interactive 3D viewer with trajectory building, confidence filtering, and time scrubbing
• Optional SAM2 integration for user-guided foreground masks
• Apache 2.0 code; model weights are CC BY-NC 4.0 (research/non-commercial only)

Caveats

• Requires ≥ 48 GB VRAM for provided examples; not exactly laptop-friendly
• Images must satisfy W ≥ H, get resized to long side 512, then cropped to multiples of 16
• Automatic downsampling kicks in if you feed it more than 40 views
• Unstructured unordered inputs are promised but not yet released

Verdict

Worth a look if you’re doing 4D reconstruction, dynamic scene understanding, or video-to-3D research and have the GPU budget. Skip it if you need commercial licensing (weights are NC-only) or want to run this on consumer hardware.