facebookresearch/large_concept_model
Meta's research on Large Concept Models, a 1.6B parameter sequence-to-sequence model that operates on semantic sentence representations in the SONAR embedding space.

This repository provides official implementations of Large Concept Models, a language modeling approach that processes sentences as semantic concepts rather than individual tokens. The model uses the multilingual SONAR embedding space supporting 200+ languages and explores multiple generation approaches including MSE regression and diffusion-based methods. The work includes training recipes for 1.6B parameter models trained on approximately 1.3T tokens, built on top of fairseq2 and PyTorch.