Dynamic data triage for LLaMA-Factory training loops

What it does DataFlex is a plug-in layer for LLaMA-Factory that turns static training data into a moving target. It implements a catalog of data-centric algorithms—sample selectors like LESS and NICE, mixture optimizers like DoReMi and ODM, and loss-based reweighting—and runs them inside the existing training loop so the model sees different data proportions or sample weights as training progresses. The project also acts as a reproducibility rescue mission for several research papers whose official code is missing or buggy.

The interesting bit Rather than treating data curation as a preprocessing step, DataFlex treats it as an optimization variable that shifts during the training loop. It is essentially high-quality glue code: it wraps fragmented academic implementations into a single registry-based system with unified YAML configuration, so you can swap selectors, mixers, and weighters without rewriting trainers.

Key highlights

• Implements hard-to-reproduce selectors (LESS, NICE, NEAR, TSDS) and mixture methods (DoReMi, ODM) under one roof.
• Drop-in compatibility with LLaMA-Factory: add DataFlex-specific YAML keys to existing configs.
• Experimental results show MMLU accuracy and perplexity improvements over static baselines on SlimPajama and Open-Hermes subsets.
• Supports DeepSpeed ZeRO-3 for gradient computation on larger models.
• Pairs with the separate DataFlow project for raw-data preprocessing, but accepts any standard JSON training format.

Caveats

• Several algorithms listed (Loss, Delta Loss, NEAR, Loss Reweighting) have no official repository, so DataFlex’s implementations are the de facto reference and should be scrutinized accordingly.
• The README flags some reproduced official repositories as problematic (⚠️), suggesting the field is still cleaning up its code.
• Python 3.11+ is recommended; Python 3.10 users need to resolve llamafactory compatibility manually.

Verdict Researchers who want to experiment with data-centric training without maintaining a patchwork of academic codebases should look here; if you are already happy with static data mixing and do not use LLaMA-Factory, this adds little value.