LLaVA gets a backbone transplant

What it does LLaVA++ retrofits the LLaVA 1.5 architecture to use Microsoft’s Phi-3 Mini Instruct (3.8B) and Meta’s LLaMA-3 Instruct (8B) as language backbones. The project publishes pretrained alignment checkpoints, LoRA adapters, fully fine-tuned weights, and S2 fine-tuned variants for both backbones through Hugging Face.

The interesting bit Instead of building a new multimodal model from scratch, the authors treat LLaVA as a chassis: swap the LLM, patch a few files, and retrain the projection layers. It is a pragmatic admission that vision encoders are sticky while language models are consumables.

Key highlights

• Ships both Phi-3-V and LLaMA-3-V with pretrained, LoRA, fully fine-tuned, and S2 fine-tuned weights on Hugging Face
• Provides live Hugging Face Space demos and a Google Colab for quick testing
• Benchmarks against standard vision-language tasks, with results shown in radar and comparison plots
• Built as a thin layer of patches and training scripts on top of the original LLaVA repository

Caveats

• It is essentially a patch set: expect to manually overlay files onto the upstream LLaVA repository before training
• README contains unedited typos (“released this weak”) and favors file-copy recipes over architectural documentation
• It is unclear whether reported improvements stem from the new backbones alone or from additional training tricks, since pretraining uses the same LCS-558K dataset as the original LLaVA

Verdict Worth a look if you are already running LLaVA and want to upgrade the language brain to LLaMA-3 or Phi-3 without rebuilding the pipeline. Skip it if you need a standalone, batteries-included framework.