Humanoids that shadow you in sim, then mimic you in reality

What it does The repository houses two complementary systems for humanoid control. The Humanoid Shadowing Transformer (HST) trains reinforcement-learning policies in NVIDIA IsaacGym to make a simulated robot track whole-body human poses. The Humanoid Imitation Transformer (HIT) handles real-world imitation learning using an ACT-style architecture, with hardware deployment code for Unitree robots on NVIDIA Jetson. The repo also points to external pose estimators—WHAM for body and HaMeR for hands—to convert human videos into motion targets.

The interesting bit Instead of hand-tuning gaits, the project treats human videos as a motion curriculum and offloads pose estimation to off-the-shelf vision models. By keeping high-frequency balance physics inside IsaacGym while delegating manipulation tasks to a real-world transformer, the codebase attempts to separate staying upright from task execution.

Key highlights

• Sim shadowing uses legged_gym and rsl_rl inside NVIDIA IsaacGym, inheriting a mature RL stack for locomotion.
• Real-world imitation uses an ACT-style transformer with ResNet-18 backbones and positional embeddings for both images and actions.
• Hardware deployment targets Unitree robots via unitree_ros2 and runs inference under JetPack on NVIDIA Jetson with PyTorch 1.11.
• The repo references external pose estimators—WHAM for body and HaMeR for hands—rather than bundling its own vision pipeline.
• A pre-collected dataset for tasks like folding clothes is available via Google Drive.

Caveats

• The README is essentially a quickstart guide; it never explains how—or whether—the HST sim policy actually transfers into the HIT real-world policy, leaving the overall architecture somewhat unclear.
• Hardware setup is tightly coupled to specific JetPack and PyTorch versions that are already archived by NVIDIA.
• The repository is described as an “updating implementation,” which suggests the code may still be in flux.

Verdict Robotics researchers working on humanoid whole-body control or sim-to-real transfer should take a look. If you are not running Unitree hardware or do not have IsaacGym set up, much of the value is read-only.