Neural thickets: a jungle of experts hiding in plain sight

What it does

RandOpt treats a pretrained model’s weight space like a coral reef: tiny random mutations around the base checkpoint spawn diverse, highly specialized “experts” for different tasks. The system uses black-box optimization (specifically, random search with selection) to discover these variants, then ensembles the top performers. It also supports distilling that ensemble back into a single model.

The interesting bit

The core claim is that competent task experts are densely packed around pretrained weights—you don’t need to train far, or even use gradients, to find them. This flips the usual narrative that fine-tuning carefully nudges weights toward a single optimum. The paper (ICML 2026 Spotlight) suggests there’s a whole ecosystem of specialists hiding in the immediate neighborhood.

Key highlights

• Operates entirely in weight space via random perturbations; no backpropagation during search
• Supports both single-node and multi-node deployment, with Slurm scripts provided
• Includes a distillation pipeline to collapse top-k experts into one deployable model
• Ships with baseline comparisons and a Colab notebook for 1D intuition-building
• Targets LLMs and LoRA adapters, with reasoning tasks among the advertised use cases

Caveats

• The README is essentially a routing layer to other markdown files; actual methodology details live in the paper and supplementary guides
• “ICML 2026” implies either a typo, a very early preprint, or time travel—sources don’t clarify which
• Docker and Slurm dependencies suggest this is built for cluster-scale experiments, not casual laptop probing

Verdict

Worth a look if you’re probing the geometry of pretrained weight spaces or building cheap ensemble methods. Skip it if you need a polished, batteries-included training framework; this is research scaffolding with the paper as the real artifact.