FastMoE provides an easy-to-use system for supporting Mixture of Experts models in PyTorch. It contains customized C and Python operators for efficient distributed training of MoE layers. The project enables extending Transformer-based models with MoE architecture through a simple plugin interface, supporting features like distributed experts and NCCL-based communication.