Your hands, now in 3D, courtesy of a transformer

What it does

HaMeR predicts a full 3D mesh of a human hand from an ordinary photograph. It outputs MANO parameters—the standard rigged hand model used in graphics and vision—so the result is not just a point cloud but an animatable mesh with joint angles. The system handles the full frame, detects the hand, and reconstructs geometry in one shot.

The interesting bit

The “transformer” in the title is doing real work here. HaMeR uses a ViT-based architecture (specifically building on ViTPose) to regress hand pose and shape directly from image tokens, rather than relying on heatmaps or part detectors as intermediate crutches. The authors also released the HInt dataset annotations, a benchmark targeting hand-hand and hand-object interactions where occlusion makes reconstruction genuinely hard.

Key highlights

• Single-image inference; no stereo or depth required
• Outputs standard MANO parameters, compatible with existing graphics pipelines
• Evaluation covers FreiHAND, HO-3D, and the newer HInt, EPIC-Kitchens, and Ego4D benchmarks—including visible and occluded splits
• Docker and Hugging Face Spaces demos available; Colab notebook provided
• Took 2nd place in the Ego-Exo4D Challenge ego-pose hands track (2024)

Caveats

• Requires downloading the MANO model separately; registration needed on the MANO website
• Training data and evaluation metadata are fetched via shell scripts and Dropbox links, not hosted in the repo itself
• Right-hand-only for MANO; left hand support is unclear from the README

Verdict

Worth a look if you need off-the-shelf hand mesh recovery for AR/VR, robotics, or human-computer interaction research. Skip it if you need real-time performance guarantees or full bimanual reconstruction—the README is silent on inference speed and left-hand handling.