Vision-language models built monolithically from pixels to text

What it does

NEO is a family of native vision-language models that process images and text inside a single dense transformer instead of using the usual pattern of a pretrained vision encoder fused with a language model. The project releases weights, training code, and evaluation kits for two model generations—NEO and NEO-ov—alongside a HuggingFace blog post describing a newer unified architecture called NEO-unify. The README states that the models learn visual perception from scratch using 390M image-text pairs and are intended to rival top-tier modular VLMs while outperforming other native approaches.

The interesting bit

Most multimodal models are modular by convenience: clip a vision encoder to an LLM and call it a day. NEO bets that a monolithic “from pixels to words” design can be competitive with far fewer training examples—390M image-text examples versus the usual scale of modular pipelines—though the README does not specify exact benchmark scores or comparison targets.

Key highlights

• Native, encoder-free architecture that unifies pixel encoding, alignment, and reasoning in one dense model
• Released training code (VLMTrainKit), evaluation scripts (VLMEvalKit and VLMEvalKit_ov), and model weights on HuggingFace for both NEO and NEO-ov
• Claims strong visual perception learned from scratch with 390M image-text examples, rivaling modular VLMs
• Roadmap includes NEO-unify, a unified multimodal architecture described in a HuggingFace blog post and a technical report for SenseNova-U1

Caveats

• Specific benchmark numbers and comparison targets are not provided in the README; performance claims are stated without supporting metrics
• The project spans multiple overlapping variants (NEO, NEO-ov, NEO-unify, SenseNova-U1) that are not clearly differentiated in the top-level documentation

Verdict

Researchers and engineers experimenting with native multimodal architectures should look here for open weights and training code; if you need drop-in replacements for production vision-language pipelines, the README offers no performance tables or API to judge readiness.