Teaching LLM committees to think in trees

What it does MARTI is a training framework that treats a network of LLM agents as a single reinforcement-learning problem. It centralizes how agents interact and share rewards while distributing the actual policy training across individual models. The newer MARTI-v2 layer adds Monte Carlo tree search to the mix, targeting tasks like code generation where multi-step exploration matters.

The interesting bit Instead of just chaining prompts, MARTI actually trains the agents’ weights through RL algorithms like GRPO and REINFORCE++, and it can simultaneously fine-tune heterogeneous models—say, a Qwen3-8B alongside a different 8B model—with independent roles and per-agent sample filtering. The tree-search variant (MARS²) swaps token-level PPO for sequence-level GSPO loss and applies a “TIS correction” to compensate for vLLM sampling drift during long rollouts.

Key highlights

• Supports graph-based workflows (debate, chain-of-agents, mixture-of-agents) and third-party integrations with AutoGen and CAMEL.
• MARTI-v2 adds single- and multi-agent MCTS training for code generation, with adaptive node expansion and refinement.
• Handles heterogeneous multi-agent training where different models run on independent strategies and dynamic sample filters.
• Incorporates sequence-level GSPO loss and TIS correction for stable training on sequences up to 32K tokens.
• Built on top of OpenRLHF and verl, using the vLLM v1 Engine and a Hybrid Engine for throughput.

Caveats

• The README lists steep hardware requirements: roughly eight 80GB GPUs for single-agent runs, and a full 8-GPU node per agent for multi-agent training.
• Third-party framework integrations (AutoGen, CAMEL) are marked as experimental.
• Reported benchmark gains are specific to the LCB code-generation dataset; broader task coverage isn’t shown in the sources.

Verdict Worth a look if you’re running a multi-agent LLM research lab and have the GPU budget to train agent teams end-to-end. If you’re looking for a lightweight CPU-friendly orchestration layer, this is not your tool.