Push every neuron, keep every rollout, see the whole model

What it does

neuron-explorer takes a layer of Qwen3.5-4B, pushes every MLP neuron up and down across a few prompts at several doses, and stores every single rollout — text, token ids, logprobs, top-k per step. A local web explorer then lays those results over a 3D perspective view of the model: one point per neuron, attention and MLP blocks per layer. You can hover for steer and logit lenses, click into a neuron, and run live steered chat with the model loaded locally on MPS, CUDA or CPU.

The interesting bit

The methodology is unusually honest about controls: neurons are ranked by word-cluster mass against random-direction controls, and the sweep itself includes 256 random controls plus unsteered baselines per prompt. Doses are defined relative to the median residual norm leaving the layer, not arbitrary magnitudes. There’s also an agent tab that runs the claude CLI against the explorer as an MCP server, so an LLM can read lenses, run steered inference, and drive the map for you.

Key highlights

• 3D map of the model with one point per neuron; writes layout clusters neurons by similar down-projection columns
• Steered rollouts per prompt and dose, plus a three-column (−, 0, +) multi-turn steered chat
• UMAP embeddings of a layer’s rollouts, colourable by neuron, prompt, dose or kind
• Per-neuron connections: the 24 strongest direct-path links up and down the network
• Sweep tooling covers stats, ranking, steer lenses, depth reports across layers, and HF-vs-vLLM equivalence checks

Caveats

• The sweep proper needs a CUDA pod and a steering fork of vLLM (RhizoNymph/vllm); the explorer’s shipped caches let you browse without running it, though
• The agent tab requires the claude CLI installed and logged in, and it gets file tools and a shell in your home folder — know what you’re granting
• Findings live in research_log.md; the README doesn’t summarize what the sweeps actually found, so expect to read

Verdict

For mechanistic-interpretability folks who want exhaustive steering data with a visual browser instead of a pile of JSONL, this is a rare treat. If you’re not already curious what individual MLP neurons do to a 4B model, the 3D map alone is still worth a look.