Turn your pile of fine-tuned LLMs into one MoE

What it does

Mergoo takes fully fine-tuned or LoRA-tuned LLM experts and weaves them into a single model using Mixture-of-Experts, Mixture-of-Adapters, or layer-wise merging. It exposes the merged result as a standard Hugging Face transformers model, so you can fine-tune just the router gates or the entire merged stack. Supported base architectures include Llama, Mistral, Phi3, and BERT.

The interesting bit

Instead of forcing you to pick one fine-tuned checkpoint, the library lets you configure which layers get routers, how many experts fire per token, and whether to restrict routing to specific decoder layer indices. For LoRA experts, it composes a Mixture-of-Adapters by building a routing layer on top of existing adapters rather than merging full dense weights.

Key highlights

• Supports MoE, Mixture-of-Adapters, and layer-wise merging from a single Python config.
• Works with Llama, Mistral, Phi3, and BERT via standard Hugging Face Trainer, SFTTrainer, and PEFT.
• Lets you train only the router gates or run full fine-tuning on the merged model.
• Saves merged checkpoints to standard Hugging Face-compatible formats, including sharded safetensors.
• Runs on CPU, MPS, or GPU.

Caveats

• Router gates start untrained, so expect weight-loading warnings on first load.
• Several conveniences—router load-balancing loss, lazy tensor loading for low-memory merging, and FlashAttention—are still on the roadmap.
• Other layer-wise merging methods and additional architectures like Gemma or Mamba are not yet supported.

Verdict

Grab this if you are sitting on multiple domain-specific fine-tunes and want to frankenstein them into an MoE without leaving the Hugging Face ecosystem. Skip it if you need a finished training stack with all the latest optimizations already wired in.