14,616 ways to describe a cartwheel

What it does

HumanML3D is a dataset and processing pipeline that marries 3D human motion capture with natural language annotations. It combines the HumanAct12 and AMASS datasets into 14,616 motion clips, each annotated with 3–4 crowd-sourced sentences covering everything from walking to acrobatics. Because AMASS redistribution is restricted, the repository provides extraction scripts that turn the source data into normalized motion vectors, joint positions, and text files with timing metadata.

The interesting bit

The authors doubled the dataset size through a simple but effective mirroring trick: they flipped all motions and swapped directional keywords in the descriptions—left becomes right, clockwise becomes counterclockwise. The annotations also include start and end timestamps, allowing annotators to caption sub-motions when a full clip is too complex to describe in one breath.

Key highlights

• 14,616 motions and 44,970 descriptions spanning 28.59 hours of activity
• Annotations cover daily actions, sports, acrobatics, and dance with an average description length of 12 words
• Includes a processed version of the KIT Motion-Language Dataset for cross-corpus experiments
• Data follows the SMPL skeleton structure (22 joints) with pre-computed mean and standard deviation files
• Underpins several text-to-motion models including T2M, TM2T, and MoMask

Caveats

• The main HumanML3D corpus is not redistributable by the authors; using it requires local access to AMASS source data, SMPL+H, and DMPL models.
• The preprocessing pipeline is version-sensitive; the README flags that matplotlib==3.3.4 is required to avoid small data drift in generated outputs.
• KIT-ML is provided directly, but the README notes it carries its own citation requirement.

Verdict

Researchers building text-conditioned motion generators should treat this as a foundational benchmark. Casual experimenters may find the multi-step assembly required to reconstruct the main dataset from external dependencies more trouble than it is worth.