Transformer model reconstructs and tracks 3D humans from casual video

What it does The project is the official implementation of an ICCV 2023 paper that reconstructs 3D human pose and shape from images and tracks them through video. It bundles a transformer-based reconstruction network with a tracking backend built on PHALP, producing mesh outputs and per-frame tracklets from sources including video files and YouTube links. Checkpoints and dependencies auto-cache on first use, though you still need to register for the SMPL body model separately.

The interesting bit The pipeline marries ViTDet detection with an HMR2.0 transformer reconstruction network, then hands the results to PHALP for temporal tracking. The authors trained the core model for seven days on eight A100 GPUs, which suggests the casual-video-to-mesh pipeline is backed by serious compute.

Key highlights

• Reconstructs 3D humans from single images and tracks them through video sequences
• Outputs .obj meshes, rendered views, and tracklets with 3D pose and shape data
• Accepts video files, frame folders, and YouTube URLs as input sources
• Provides both a released checkpoint (HMR2.0b) and an alternative (HMR2.0a) for evaluation
• Includes preprocessing utilities for standard benchmarks like Human3.6M, 3DPW, and COCO

Caveats

• Requires a separate manual download and registration for the SMPL neutral model
• Video tracking depends on an external PHALP installation and, for YouTube links, on pytube working properly
• Training from scratch demands eight A100 GPUs and a week of compute time

Verdict Worth a look if you need off-the-shelf 3D human reconstruction and tracking for research or prototyping. Give it a pass if you need a lightweight, SMPL-free, or fully self-contained pipeline.