The 3B overachiever built on déjà vu data

What it does Stability AI uses this repository to publish its StableLM family of open language models, currently anchored by the 3-billion-parameter StableLM-3B-4E1T. It is a decoder-only transformer trained on one trillion tokens run through four epochs—an intentional stress test of whether repeating data erodes downstream performance. The repo also catalogs earlier Alpha base and tuned checkpoints, plus StableVicuna-13B, an RLHF offshoot of Vicuna.

The interesting bit The project treats the multi-epoch regime as a feature, not a bug. StableLM-3B-4E1T uses LayerNorm with learned bias instead of the now-ubiquitous RMSNorm, applies rotary position embeddings to only the first quarter of head dimensions for throughput gains, and somehow outscores Stability AI’s own 7B Alpha v2 on zero-shot benchmarks—suggesting that compute budget and data repetition can sometimes substitute for raw parameter count.

Key highlights

• StableLM-3B-4E1T claims state-of-the-art (as of September 2023) among open 3B-parameter models and sits competitively in the 7B bracket on standard zero-shot tasks.
• Architecture follows LLaMA but swaps in LayerNorm, partial rotary embeddings, and a GPT-NeoX tokenizer.
• Training mix draws from Falcon RefinedWeb, RedPajama, The Pile (minus Books3), and StarCoder.
• Base checkpoints are under CC BY-SA-4.0; StableVicuna-13B delta weights are CC BY-NC-SA-4.0 due to LLaMA lineage.
• YAML configs and raw evaluation JSONs live in the repo for reproducibility.

Caveats

• The README warns that the base model contains a lot of web data and should be fine-tuned before serious downstream use.
• StableVicuna-13B inherits LLaMA’s non-commercial licensing restrictions.
• The repository is essentially a model card and checkpoint index; training code and infrastructure are not provided.

Verdict Bookmark it if you’re probing small-model scaling laws or need a permissively licensed 3B base to fine-tune. Look elsewhere if you want a drop-in chat model or a complete training framework.